How to identify disk abusing process?

[cjb at mrao.cam.ac.uk (Chris Ball)]

Bob,

   > So here is a Linux question that's been nagging at me.  Is there a
   > simple (command line) way of identifying processes that are
   > creating heavy disk i/o?  I suppose I could wrap some shell script
   > around sar, or poke around in /proc, but thought asking here might
   > be more productive and fun.  Any thoughts?

This sort of thing is a fun use of systemtap, which is the Linux
equivalent of Sun's dtrace.  Here's a quick script to count calls
to vfs_{read,write} made via each process:

unity:cjb~ % cat listio.stap
global execnames

probe kernel.function("vfs_write"), kernel.function("vfs_read") {
        execnames[execname()]++
}

# print top processnames every 5 seconds
probe timer.ms(5000) {
        foreach ([name] in execnames-) {
                # putting the - after execnames means sort descending
                printf("proc: %s, count: %d\n", name, execnames[name])
        }
        delete execnames
}

unity:cjb~ % sudo stap listio.stap
proc: Xorg, count: 896
proc: kiba-dock, count: 250
proc: compiz, count: 184
proc: gnome-terminal, count: 124
proc: gai-temp, count: 81
proc: cpufreq-applet, count: 76
proc: gnome-power-man, count: 60
proc: mixer_applet2, count: 50
proc: ssh, count: 50
[..]

   > A secondary, less important, question is how to identify the
   > file(s) being accessed.

Something like:

unity:cjb~ % cat whichfiles.stap
probe kernel.function("sys_open") {
  if ($flags & 1) {
    print(execname() . " writes " . user_string($filename) . "\n")
  } else {
    print(execname() . " reads " . user_string($filename) . "\n")
  }
} 

unity:cjb~ % sudo stap whichfiles.stap
hald-addon-stor reads /dev/hdc
cpufreq-applet reads /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
cpufreq-applet reads /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
cpufreq-applet reads /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cpufreq-applet reads /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
cpufreq-applet reads /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed
cpufreq-applet reads /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
sendmail reads /proc/loadavg
gai-temp reads /proc/acpi/thermal_zone/THRM/temperature
gai-temp reads /sys/devices/platform
NetworkManager reads /sys/class/net/eth0/carrier
[..]

I blogged about another use of systemtap a few months ago:

   http://blog.printf.net/articles/2006/05/11/systemtap-for-fun-and-profit

- Chris.
-- 
Chris Ball   <cjb at mrao.cam.ac.uk>    <http://blog.printf.net/>