Can I haz ur dataz - Fri, Feb 1, 2019
Remote data acquisitions over ssh
To get your data trough
ssh to your local storage, you simply use pipes. It does not matter if you use
dd or any other command line tool which outputs the data on standard output (stdout).
When using cat the there is no need to add any additional parameters in your command chain. A simple
cat <input> will suffice.
Cat does not have any progress information during read operations.
dd requires more user interaction during the setup phase. To use
dd you have to give the if= argument. The use of different blocksizes (bs) will not have that much of an impact on the speed. Feel free to have a look at this this.
Wow. So scientific! Much CLI.
A simple example with no output to the terminal except of errors. The output to stderr is not suppressed.
$ # Using cat to copy /dev/sda $ ssh <user@remotehost> 'cat /dev/sda' > sda.raw
If you want to suppress errors, use:
$ # Using cat to copy /dev/sda $ ssh <user@remotehost> 'cat /dev/sda 2>/dev/null' > sda.raw
The next example will demonstrate the use of
$ # Using dd to copy /dev/sda $ ssh <user@remotehost> 'dd if=/dev/sda' > sda.raw
Of course you can suppress errors as well.
Speeding up the data acquisition and minor tweaks
With the basics covered, we can begin optimizing our data transfer. In the first step we speed up the transfer with
The argument -c will write the compressed data to stdout which will be piped to your local system.
Of course you can use this with
cat as well.
$ ssh <user@remotehost> 'dd if=/dev/sda | gzip -c' | gunzip > sda.raw
To add some progress information, you have two options with
$ # dd status $ ssh <user@remotehost> 'dd if=/dev/sda status=progress | gzip -c' \ > | gunzip > sda.raw
The status argument writes the output to stderr and will not end up in your local copy.
$ # dd through pv $ ssh <user@remotehost> 'dd if=/dev/sda | gzip -c' \ > | pv | gunzip > sda.raw
Pv needs to be installed separatly. Check your packet manager. 0r teh Googlez!
Fixed a problem within the examples. Had a pipe too much. #Fail