UDR Transport Utility

Today I’d like to go over a utility that I’ve been exposed to that has changed the way I move large files between systems. It is called UDR. UDR is a wrapper for UDT; which is short for UDP-based Data Transfer, utility. As you may have guessed, UDT uses the UDP protocol to transfer large files, and it does so really, really fast.

The problem is that up to now, the best way that I’ve seen to transfer large files is to run multiple rsync instances in order to fill the amount of bandwidth available. A single instance of rsync is simply too high overhead to transfer a large file very fast. Of course its not rsync’s fault. From what I’ve seen the problem more lies on the fact that rsync utilizes TCP which has characteristics which in high bandwidth/high performance situations cause it to not perform well.

As an example of the performance difference, when transferring a file using rsync from one server to another with a 1Gb link between, I was unable to send faster than 160 Mbps. Sending the same file using UDT though, and it transferred at 900 Mbps. Watching where the load was on the machines during the transfer made it seem that UDT could have actually sent the file faster, except the receiving server was using a single rotational disk and could only write at 112 MBps.

The best part is that UDT has a wrapper as I mentioned called UDR. UDR uses the UDT utility to transfer files, but then uses rsync to verify they transferred correctly. And the setup and syntax are really easy to understand if you’re used to using rsync. Keep in mind that rsync uses SSH, so you’ll be asked for login information for the destination server unless you’re using SSH keys. For my example though, the command looked similar to:

udr rsync -av --progress /path/to/file dest_server:/dest/path/

Now UDR does support some options of it’s own, such as encryption, timeout value, and ports to use . See the help for more details.

For more information on UDT: http://udt.sourceforge.net/doc.html

For more information on UDR: https://github.com/LabAdvComp/UDR

And for a Salt State to get this up and running: https://github.com/alektant/salt_states/tree/master/udr

Originally published at https://smartaleksolutions.com on November 24, 2015.