In setting up some mirrors recently, I’ve come to learn that rsync’s algorithm by default doesn’t deal well with long-haul TCP going hand in hand with small files. I still need to set aside some time to find a nice set of optimization flags to tweak that up a bit more. And when I say “doesn’t deal well”, I mean in the region of “can tx about 1/3rd a single-session TCP speedtest can do over the same path”. Later’s worry, though.
So, the point of this point:
receiver$ nc -l 1234 | pv | tar xf
sender$ tar cf - moz | pv | nc receiver.domain.tld 1234
And there you have a minimal-CPU-usage streaming file delivery setup. Of course, this doesn’t deal with retransmits or anything else. This just gets the bulk over. After that you can run an rsync with big block lengths over all of it, and get any files fixed using that.
And the “old trick” part of this? As far as I know, this is pretty much how the whole tape drive thing used to work[0]. I’ve just added some internet in the middle.
[0] – I wouldn’t know for certain, before my time