[Discuss] Help with data migration

[invalid at pizzashack.org (Derek Martin)]

On Mon, May 27, 2013 at 10:08:39AM -0400, Greg Rundlett (freephile) wrote:
> In the past, on boxes I've setup, rsync just flies.  In this case, we've
> got 10gE cards and robust hardware, quad core, 16GB RAM in the VM instance
> and VMWare ESX running on the iron, and something is just not right.  In
> summary, after NFS mounting the source, and switching to cp, I'm getting
> speeds as slow as 2.2MB/s for transfer.  rsync "locally" is only doing
> about 9MB/s.  I'm not sure how to check where the bottleneck is or what's
> wrong.
 
Do you mean locally as in rsync over NFS?

Here are a few suggestions:

NFS by default uses (or at least used to) a transfer buffer size of
4k, which produces crappy performance.  These days it's supposed to
negotiate between client and server; you can find the size which was
negotiated by looking in /proc/mounts.  If it's small (4k is the
minimum), try setting the rsize and wsize options in your mount
command (or fstab) to request a larger size.  You might do better with
something like 16k, 64k, or even larger, up to a max of 1MB.

In the Bad Old Days, having a network card duplex mismatch was a
common problem that could monumentally impact network performance.
I haven't really seen issues with this in years, but it might be
something to rule out.  You can use mii-tool or ethertool to check to
make sure your card has negotiated the correct speeds/duplex.

For rsync performance, you'd said you enabled compression.  Turn it
off; as has been said earlier in the thread, it may well be faster to
send the uncompressed bytes.  You can also try having SSH use the
blowfish protocol, which I believe is the fastest encryption algorithm
it supports.  It used to have an option to disable encryption
entirely, but that was removed AFAIK.

If none of that does the job, you can try using the 4 horsemen of
performance monitoring to see what's going on:  top, sar, vmstat, and
iostat.  I won't go into details on how to use them as that would be
the equivalent of writing a chapter of a book. ;-)  But you can find
resources in their man pages and on line.  You can also strace/ltrace
your copy process (whatever it is) to see what it's doing, which may
give you some idea of how its (mis)spending its time.

Good luck.

-- 
Derek D. Martin    http://www.pizzashack.org/   GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.  Replying to it will result in
undeliverable mail due to spam prevention.  Sorry for the inconvenience.