如果使用设备块大小，则 mbuffer 比 dd 慢

Question

I have an backup script which is generating a tar on the fly and piping it to mbuffer. The data is then written to a tape drive (LTO3)

I discovered that the mbuffer slows the bandwidth down and I can not figure out why.

here the 2 commands and the average speeds below them

$tar -b 512 -cf - /data | \
 mbuffer -A "..." -P 90 -m 1G -f -o /dev/st0 -d 512

in @ 21.8 MB/s, out @ 21.8 MB/s, 1287 MB total, buffer 100% full

and if I pipe the data again to dd it will result in a much higher bandwidth

$tar -b 512 -cf - /nas/homes/ /nas/photo/ | \
 mbuffer -P 90 -m 1G | \
 dd of=/dev/st0 bs=256k

in @ 72.9 MB/s, out @ 64.0 MB/s, 2671 MB total, buffer  99% full

My question is if I am using mbuffer the wrong way or if its not supposed to be used with -d . Even if I'm not specifying the block-size with -d the speed stays the same.

I would like to use mbuffer because of the -A flag but with this performance it would take triple the time.

Answer 1

Well, from the mbuffer(1) man page:

-s <size>
           Use blocks of size bytes for buffer (default is determined on startup).

But, I think the key is that in your second command, you're using 262144 byte (256k) writes and only 512 byte writes in the first. I suspect if you changed the second command to bs=512 and the first command to -s 262144 you'd invert the situation.

It'd be ideal if they matched. So, try -s 262144 with mbuffer.

Edited to reflect a correction in the David's answer.

Answer 2

-d isn't an option you can specify a parameter; it's used to grab the default size. To over-ride I recommend you remove the -d and use -s in bytes

eg

-s 512000

or use 1024*512 = 524288

Answer 3

I believe LTO uses a block size minimum of 512, not 256 if memory serves.Anything less is extremely inefficient. Try an 8mb minimum buffer- youll get better results.

Using dd incorrectly throws about 3 different errors with these drives, especially during tape restore. Document your methods used. It will save you hours of headache later on.

'mt /dev/ setblk 512' can be used to overcome this(mt-st package)- however- uses for the variable block size exist, such as in using dump/restore(ext4 works best) or Disk archive(as opposed to Tape archive).

Lzip will not split- plan accordingly.Its fine if your data "should" otherwise fit.

There seems to be a lack of "long term data safe" storage methods. Tar isnt what it was once cracked up to be. It was fine for reel2reel- not so much anymore. Theres no "integrity" or means to skip over bad files these days- and the methods that can do so- often are not used--or are a real PITA to implement/setup post OS install. There is no "it just works" solution, unfortunately. Thats the nature of tape for ya. Disk archive(DAR) is a decent replacemnt for Tar- but be patient with it- DO NOT screw up header writes on WORM tape- its a real pain to fix on recovery.

LTFS just makes things worse, not better. Dont jump to LTO4+ just for the "filesystem". Its a pain to get going (correctly) on linux.