sendmail and spambots

[pbaumgardner-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org (Paul Baumgardner)]

Hello everyone,

I'm hoping there are some sendmail/email wizards that could offer some
advice/comments on my situation.

I'm running 4 sendmail servers that process approximately 3 million
transactions per day.  I'm running Sendmail 8.13.8-2.el5 on CentOS
5.2.  I'm running powerdns-recursor on each server for DNS queries,
using RBL's from spamhaus and TrendMicro to block IP's, and ClamAV.
Using the RBL's, I'm blocking about 1.5 million SMTP connections per
day.

In recent months I've noticed a spike in the number of sendmail
processes.  The spike seems to be a result of spambots slamming my
servers with connections.  The IP's are getting blocked by the RBL's,
however the connections are sitting around for a short time and I
guess there are just so many.  I'm using greet_pause and have it set
to 5000 (5 sec).  I also have some pretty aggressive timeouts set:

define(`confTO_IDENT', `0s')
define(`confTO_CONNECT', `30s')
define(`confTO_INITIAL', `30s')
define(`confTO_ACONNECT', `1m')
define(`confTO_HELO', `30s')
define(`confTO_RCPT', `30s')
define(`confTO_RSET', `30s')
define(`confTO_QUIT', `30s')
define(`confTO_MISC', `30s')
define(`confTO_COMMAND', `30s')
define(`confTO_CONTROL', `30s')
define(`confTO_ICONNECT',`30s')
define(`confTO_MAIL',`30s')
define(`confTO_DATAINIT',`30s')
define(`confTO_DATABLOCK',`1m')
define(`confTO_DATAFINAL',`1m')
define(`confTO_STARTTLS',`1m')
define(`confTO_AUTH',`1m')
define(`confCONNECTION_RATE_THROTTLE', `30')
define(`confMAX_MESSAGE_SIZE',`12485760')
define(`confLOG_LEVEL', `12')
define(`confTO_QUEUERETURN',`2d')
define(`confREFUSE_LA',`12')
define(`confQUEUE_LA',`100')
FEATURE(`greet_pause',5000)dnl

Normally, each server is running about 20-30 sendmail processes.
Usually twice per day, for about 2 hours each time the process count
for sendmail will spike to around 200.

If I do a 'ps aux | grep sendmail'  I see a lot of processes sitting
in cmd read state.  Here's a sample:

root      9248  0.0  0.0  73976  4424 ?        S    12:23   0:00
sendmail: server [212.57.148.218] cmd read
root      9302  0.0  0.0  73712  3464 ?        S    12:23   0:00
sendmail: server 201-88-214-86.cbace700.dsl.brasiltelecom.net.br
[201.88.214.86] cmd read
root      9303  0.0  0.0  73704  3452 ?        S    12:23   0:00
sendmail: server 189104163027.user.veloxzone.com.br [189.104.163.27]
(may be forged) cmd read
root      9335  0.0  0.0  73704  3464 ?        S    12:23   0:00
sendmail: server 200-96-168-116.bsace705.dsl.brasiltelecom.net.br
[200.96.168.116] cmd read
root      9350  0.0  0.0  73708  3452 ?        S    12:23   0:00
sendmail: server abs-static-188.8.68.58.aircel.co.in [58.68.8.188]
(may be forged) cmd read

I also see lots of entries like this in my maillog:

Oct  1 07:41:51 mailserver1 sendmail[777]: n91CfEkK000777: timeout
waiting for input from [119.152.57.66] during server cmd read

and

Oct  1 16:07:35 mailserver1 sendmail[22315]: deferring connections on
daemon MTA: 30 per second

Oct  1 07:41:51 mailserver1 sendmail[777]: n91CfEkK000777: timeout
waiting for input from [119.152.57.66] during server cmd read

Sep  9 10:22:19 mailserver1 sendmail[30162]: n89FMJNV030162: rejecting
commands from 201-40-148-66.paemt701.dsl.brasiltelecom.net.br
[201.40.148.66] due to pre-greeting traffic


Which I know is a result of having set CONNECTION_RATE_THROTTLE to 30.

I also notice that when these spikes happen, I have a lot of
connections sitting in SYN_RECV state on each server, there's usually
hundreds.  Here's a few:

netstat -tn | grep SYN_RECV

tcp        0      0 10.10.1.1:25              189.105.231.63:63034
   SYN_RECV
tcp        0      0 10.10.1.1:25              189.74.16.49:42481
   SYN_RECV
tcp        0      0 10.10.1.1:25              201.15.66.43:27890
   SYN_RECV


Can anyone offer some advice or suggestion on troubleshooting?
Something else I should be looking at or doing?  I'm kind of learning
as I go with sendmail and don't have a ton of experience with high
volume mail servers.

Thanks for any suggestions.