~mbuffer/
home of measuring buffer

Description

mbuffer is a tool for buffering data streams with a large set of unique features:

direct support for TCP based network targets (IPv4 and IPv6)
ability to send to multiple targets in parallel (distribution mode)
support for multiple volumes
I/O rate limitation
high/low watermark based restart criteria
configurable buffer size
on the fly MD5 hash calculation
highly efficient, multi-threaded implementation

It is especially useful, if you are writing backups to fast tape drives or libraries. Those drives tend to stop and rewind if they have a buffer underrun. This so called tape screwing reduces the lifetime of the motors.
mbuffer can prevent buffer underruns, if used correctly and speed up the whole backup process. Please read the man page for details, how to use it.

Technology/Features

mbuffer is written in C and uses the POSIX API. Initially, I developed it under Linux to learn multi-threaded programming and speed up my own backup process, which used a DAT tape. Later it got enhanced features like network support. Today I develop mbuffer on Solaris and only use it from time to time to mirror some diretory tree from one machine to another.

Usage/Command line options

-b <num> : use <num> blocks for buffer (default: 11326)
-s <size> : use block of <size> bytes for buffer (default: 8192)
-m <size> : memory <size> of buffer in b,k,M,G,% (default: 2% = 88M)
-L : lock buffer in memory (unusable with file based buffers)
-d : use blocksize of device for output
-D <size> : assumed output device size (default: infinite/auto-detect)
-P <num> : start writing after buffer has been filled more than <num>%
-p <num> : start reading after buffer has been filled less than <num>%
-i <file> : use <file> for input
-o <file> : use <file> for output (this option can be passed MULTIPLE times)
-I <h>:<p> : use network port <port> as input, allow only host <h> to connect
-I <p> : use network port <port> as input
-O <h>:<p> : output data to host <h> and port <p> (MUTLIPLE outputs supported)
-n <num> : <num> volumes for input
-t : use memory mapped temporary file (for huge buffer)
-T <file> : as -t but uses <file> as buffer
-l <file> : use <file> for logging messages
-u <num> : pause <num> milliseconds after each write
-r <rate> : limit read rate to <rate> B/s, where <rate> can be given in b,k,M,G
-R <rate> : same as -r for writing; use eiter one, if your tape is too fast
-f : overwrite existing files
-a <time> : autoloader which needs <time> seconds to reload
-A <cmd> : issue command <cmd> to request new volume
-v <level> : set verbose level to <level> (valid values are 0..5)
-q : quiet - do not display the status on stderr
-c : write with synchronous data integrity support
-H
--md5 : generate md5 hash of transfered data
-4 : force use of IPv4
-6 : force use of IPv6
-0 : use IPv4 or IPv6
-V
--version : print version information

Download

Binaries

OpenCSW Package for Solaris

Sources

LICENSE:

GNU GPL 3

DESCRIPTION:

mbuffer is a replacement for buffer with additional functionality:

display of I/O speed
optional use of memory mapped I/O for huge buffers
multithreaded instead of sharedmemory ipc
multi volume support
autoloader support
networking support
compatible command-line options

TESTING:

to test the program I run an integrity check with gpg

ARCHITECTURES:

This software has been successfully tested under the following operatingsystems:

Solaris (primary target)
FreeBSD
Linux
Tru64

Thomas Maier-Komor

~mbuffer/ home of measuring buffer