SUMMARY:find the PID doing DNS queries?

[dkressin-/E1597aS9LQAvxtiuMwx3w at public.gmane.org (Dan Kressin)]

I asked about this a while back and then it got backburnered until yesterday.

PROBLEM:
  How to determine which PIDs on a RHEL3 (2.4 kernel) system were doing lookups against our legacy DNS server.  resolv.conf was being ignored.

SOLUTION:
 1) Set up iptables logging rules on a per-PID basis for traffic destined for the legacy DNS server:

ps -ef | awk '{print $2}' |grep -v PID | xargs -iXX iptables -A OUTPUT -d $LEGACY_DNS_IP -j LOG --log-prefix pid-XX -m owner --pid-owner XX

 2) Monitor /var/log/messages for iptables logs and parse out PIDs and process names:

awk '/pid-/ {print $6}' /var/log/messages | sort | uniq | sed -e 's/IN=//' | cut -f2 -d- | xargs ps -p

Caveats:
 1) We had no existing iptables rules, so removing the pid logging when I was done was as easy as "/etc/init.d/iptables stop".  This may be more involved with pre-existing rules.
 2) PID matching was apparently taken out in 2.6.24 due to it being "unfixable broken and stands in the way of locking changes to tasklist_lock." (from changelog)

Thanks to all for their suggestions and to Ben Eisenbraun for the iptables idea.

-Dan