Linux kernel tuning TCP/Memcached

[tbs-Gb/NUjX2UK8 at public.gmane.org (theBlueSage)]

Hi folks,

So I have been pushing the limits of my web servers and came across a
setting that made everything fabulous! :) However I cant understand why
it would be off by default in Linux, and therefore wondered if anyone
knew of a reason _not_ to use the setting. 

Here is the story ...

I have a server running CentOS apache/PHP and memcached. When pushed
under load (stress testing with JMeter) I could reliably break my setup.
I have machines capable of handling 100+Million request a day, but I am
getting in the area of 3% or that. To cut a long story short, it appears
that PHP's TCP connections to the memcached instance were each on single
TCP threads, and they started to stack up, i.e. when PHP had finished
with them it left them in a state of FIN_WAIT. When I pushed towards
300,000 requests in a minute, Netstat showed me my threads for that port
in FIN_WAIT > ridiculous .... (memcache here is running on port 11411)


> [root at api01 ~]# netstat -tulnap |grep 11411 |wc -l
> 28329


 eventually this causes the memcache connections to fail, apache gets
upset, PHP gets nervous and memcached leaves the room. and I get zero
results. After a lot of digging/reading I found the following 3 setting
being suggested by a linux tuning website :

/sbin/sysctl -w net.ipv4.tcp_tw_recycle=1 
/sbin/sysctl -w net.ipv4.tcp_tw_reuse=1
/sbin/sysctl -w net.ipv4.tcp_fin_timeout=10

now, I am familiar with the tcp_fin_timeout, but it had no affect on the
performance. Nor did 'reuse'. However, 'recycle' was incredible. The
performance rocked ( I hit the 300,000 target without problem, and my
netstat result :


> [root at api01 ipv4]# netstat -tulnap |grep 11411 |wc -l
> 11
> 

it never went higher than 11.


I have since reset the tcp_fin_timeout to something more reasonable (30)
and also discovered that the 'reuse' option did not do much of anything.
IT is all on the recycle ..

So .... does anyone have any experience with this setting under load?
does it have any effect (good/bad) on apache or NFS? I have read 'the
interwebs' but wondered if there were any more personal explanations of
why it might or might not be a good idea.

specs and goods on my server :

Server : 4 x Quad Intels, 26G RAM
kernel : 2.6.18-164
CentOS v 5.4 
Apache 2.2.14
Php : 5.2.11
Memcached 1.4.4


thanks for reading this, any suggestions welcome :)

Richard
 



-- 
theBlueSage <tbs-Gb/NUjX2UK8 at public.gmane.org>