Uncategorized

Can I haz ur dataz

Posted on by jpk

Remote data acquisitions over ssh

To get your data trough ssh to your local storage, you simply use pipes. It does not matter if you use cat, dd or any other command line tool which outputs the data on standard output (stdout).

Using cat

When using cat the there is no need to add any additional parameters in your command chain. A simple cat <input> will suffice.

Cat does not have any progress information during read operations.

Using dd

The program dd requires more user interaction during the setup phase. To use dd you have to give the if= argument. The use of different blocksizes (bs) will not have that much of an impact on the speed. Feel free to have a look at this this.

Wow. So scientific! Much CLI.

Examples

A simple example with no output to the terminal except of errors. The output to stderr is not suppressed.

 $ # Using cat to copy /dev/sda
 $ ssh <user@remotehost> 'cat /dev/sda' > sda.raw

If you want to suppress errors, use:

 $ # Using cat to copy /dev/sda
 $ ssh <user@remotehost> 'cat /dev/sda 2>/dev/null' > sda.raw

The next example will demonstrate the use of dd.

 $ # Using dd to copy /dev/sda
 $ ssh <user@remotehost> 'dd if=/dev/sda' > sda.raw

Of course you can suppress errors as well.

Speeding up the data acquisition and minor tweaks

With the basics covered, we can begin optimizing our data transfer. In the first step we speed up the transfer with gzip.

The argument -c will write the compressed data to stdout which will be piped to your local system.

Of course you can use this with cat as well.

 $ ssh <user@remotehost> 'dd if=/dev/sda | gzip -c' | gunzip > sda.raw

To add some progress information, you have two options with dd.

 $ # dd status
 $ ssh <user@remotehost> 'dd if=/dev/sda status=progress | gzip -c' \
 > | gunzip > sda.raw

The status argument writes the output to stderr and will not end up in your local copy.

 $ # dd through pv
 $ ssh <user@remotehost> 'dd if=/dev/sda | gzip -c' \
 > | pv | gunzip > sda.raw

Pv needs to be installed separatly. Check your packet manager. 0r teh Googlez!

KTHXBYE.

Update #01

Fixed a problem with in the examples. Had a pipe too much. #Fail

What if the cult of dd is right?

Posted on by jpk

Are you a believer?

There are articles out there talking about the useless usage of dd and why cat is better. Cat is faster because it automatically adjusts the blocksize and dd is slow because it internally only works with 512 byte blocks. This and That.

I did some simple tests with time, dd and cat, added some obscure parameters to dd, because cat is better.

Testing dd with status and specific blocksize

 $ time dd if=/dev/sdd of=test.dd status=progress bs=8M
 15921577984 bytes (16 GB, 15 GiB) copied, 878 s, 18.1 MB/s 
 1899+1 records in
 1899+1 records out
 15931539456 bytes (16 GB, 15 GiB) copied, 879.018 s, 18.1 MB/s

 0.04s user 23.33s system 2% cpu 14:39.03 total

Testing dd

 $ dd if=/dev/sdd of=test.dd 
 31116288+0 records in
 31116288+0 records out
 15931539456 bytes (16 GB, 15 GiB) copied, 869.783 s, 18.3 MB/s
 16.13s user 159.22s system 20% cpu 14:29.80 total

Testingcat with pv

 $ time cat /dev/sdd | pv > test.raw
 14.8GiB 0:14:43 [17.2MiB/s] [        <=>                            ]
 0.28s user 25.84s system 2% cpu 14:43.18 total

Testing cat

 $ time dd if=/dev/sdd of=test.dd status=progress bs=8M
 15921577984 bytes (16 GB, 15 GiB) copied, 878 s, 18.1 MB/s 
 1899+1 records in
 1899+1 records out
 15931539456 bytes (16 GB, 15 GiB) copied, 879.018 s, 18.1 MB/s

 0.04s user 23.33s system 2% cpu 14:39.03 total

Testing dd

 $ dd if=/dev/sdd of=test.dd 
 31116288+0 records in
 31116288+0 records out
 15931539456 bytes (16 GB, 15 GiB) copied, 869.783 s, 18.3 MB/s
 16.13s user 159.22s system 20% cpu 14:29.80 total

Testing catwith pv

 $ time cat /dev/sdd | pv > test.raw
 14.8GiB 0:14:43 [17.2MiB/s] [        <=>                            ]
 0.28s user 25.84s system 2% cpu 14:43.18 total

Testing cat

 $ time cat /dev/sdd > test.raw
 0.18s user 21.21s system 2% cpu 14:42.25 total

Y U DO IT WRONG

Somehow my cat is not as fast as dd.

KTHBYE

The MQTT Broker Misquitto has some problems with unauthenticated clients

Posted on by jpk

I use Mosquitto as my main MQTT broker for my smart home environment. Recently I came across some behavior I should investigate further in near future.

The MQTT clients did not authenticate to broker with a correct password. But the clients were accepted, at least for a short period of time. After that the clients were disconnected. During this short time, the devices could receive and sent messages without being authenticated. hm…..