Idea
A fast and unencrypted data transfer for local network usage. The idea is to send a tar
stream over TCP by using the nc
tool.
The pipe viewer (pv
) is optional but allows tracking the transfer rates and total data transmitted. By comparing with the original data sizes, this can act as a progress indicator.
Local Data Copying with Tar
tar -C SOURCE-DIR -c . | pv | tar -C DEST-DIR -x
Network Data Copying
To copy data as-is, be sure to invoke the tar
commands as root such that they can read and write all data including permissions etc.
The --numeric-owner
flag is useful to transfer data as-is even on systems with different user and group IDs.
Sender Side
The sender acts as a client:
tar -C SOURCE-DIR --numeric-owner -c . | pv | \
nc -q 10 192.168.2.1 2244
Receiver Side
The receiver acts as a server on address 192.168.2.1 and port 2244. Change as required.
nc -w 10 -l 192.168.2.1 2244 | pv | tar -C DEST-DIR --numeric-owner -x
Verifying Integrity
To check if the file contents match on both ends, compute sha256sum
and compare it on the target. Note that if the paths are not equal on source and destination, it may be required to edit checksums.sha256
before using it on the receiver side.
Sender Side
find SOURCE-DIR -type f -exec sha256sum {} + | tee checksums.sha256
Receiver Side
sha256sum -c checksums.sha256 | tee checksum-results.txt
grep -v OK checksum-results.txt
Data Copying and Checksum Computation in One
Requires shell capable of process substitution.
Sender Side
tar -C SOURCE-DIR -c . | pv | tee >(sha256sum > sha256-sent.txt) | \
nc -q 10 192.168.2.1 2244
Receiver Side
nc -w 10 -l 192.168.2.1 2244 | pv | \
tee >(sha256sum > sha256-recv.txt) | tar -C DEST-DIR -x
Afterwards, compare contents of sha256-sent.txt
and sha256-recv.txt
. They should be identical!
See Also
nc(1), pv(1), rsync(1), tar(1), tee(1)
Article with similar basic ideas: