[Discuss] rsync v. cp in data migration

[invalid at pizzashack.org (Derek Martin)]

On Fri, May 24, 2013 at 06:46:48PM -0400, Richard Pieri wrote:
> Try this (as root) instead of the dd trick:
> 
>   sync && echo 3 > /proc/sys/vm/drop_caches
> 
> This will force the kernel to flush its buffers then drop the various
> caches and free the associated RAM. The caches will fill up again so
> you'll need to do this before each test to clean things out.

The reason I didn't do this (I didn't know how, but it did occur to me
it was probably possible, and google would tell me) is because I
figured it was a bit unfair...  though actually, it might make for a
more accurate test, in some sense.  Normally, when you go to do your
system will have stuff in buffer cache, and that will affect the
performance of other I/O that occurs (like your data migration) as the
kernel decides whether it needs to evict pages or not, etc..  Having
buffer cache full of data that's not relevant to your migration is
more realistic.  Having empty buffer cache means the kernel can just
add the page with no overhead.  

Though, perhaps the overhead is negligible since it may be very small,
and *probably* should not affect the performance of cp vs. tar
differently.  

> Here's another thing that might be relevant. tar is a bit of a CPU hog,
> and the tar pipe trick invokes two separate tar processes which can make
> it more CPU bound than cp -r.
> 
> One last thing: lots of small files. Thousands. Try using /usr as a source.

Got that covered already.  My tmp has compiled sources for a few
different projects in it, as well as a whole bunch of other random crap.
Thousands of small files.  Time to clean up I think. :)

-- 
Derek D. Martin    http://www.pizzashack.org/   GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.  Replying to it will result in
undeliverable mail due to spam prevention.  Sorry for the inconvenience.