Idea
A fast and unencrypted data transfer for local network usage. The
idea is to send a tar
stream over TCP by using the
nc
tool.
The pipe viewer (pv
) is optional but allows tracking the
transfer rates and total data transmitted. By comparing with the
original data sizes, this can act as a progress indicator.
Local Data Copying with Tar
tar -C SOURCE-DIR -c . | pv | tar -C DEST-DIR -x
Network Data Copying
To copy data as-is, be sure to invoke the tar
commands
as root such that they can read and write all data including permissions
etc.
The --numeric-owner
flag is useful to transfer data
as-is even on systems with different user and group IDs.
Sender Side
The sender acts as a client:
tar -C SOURCE-DIR --numeric-owner -c . | pv | \
nc -q 10 192.168.2.1 2244
Receiver Side
The receiver acts as a server on address 192.168.2.1 and port 2244. Change as required.
nc -w 10 -l 192.168.2.1 2244 | pv | tar -C DEST-DIR --numeric-owner -x
Verifying Integrity
To check if the file contents match on both ends, compute
sha256sum
and compare it on the target. Note that if the
paths are not equal on source and destination, it may be required to
edit checksums.sha256
before using it on the receiver
side.
Sender Side
find SOURCE-DIR -type f -exec sha256sum {} + | tee checksums.sha256
Receiver Side
sha256sum -c checksums.sha256 | tee checksum-results.txt
grep -v OK checksum-results.txt
Data Copying and Checksum Computation in One
Requires shell capable of process substitution.
Sender Side
tar -C SOURCE-DIR -c . | pv | tee >(sha256sum > sha256-sent.txt) | \
nc -q 10 192.168.2.1 2244
Receiver Side
nc -w 10 -l 192.168.2.1 2244 | pv | \
tee >(sha256sum > sha256-recv.txt) | tar -C DEST-DIR -x
Afterwards, compare contents of sha256-sent.txt
and
sha256-recv.txt
. They should be identical!
See Also
nc(1), pv(1), rsync(1), tar(1), tee(1)
Article with similar basic ideas: