[clug] finding source of high wait states

[jm]

I've got a machine which is suffering from high CPU wait states. The 
load on the machine is sitting around 15 and it's spending about 60-90% 
of time in the wait state. This is normally assoicated with I/O of some 
form. I'm guessing disk. Is there someway to track down which program(s) 
are causing this, which files are being accessed, is there any i/o 
errors (checked logs including dmesg and don't see any), etc?

Normally I use iostat, but I came across dstat recently. Here's example 
output for this machine:

$dstat -D total,sda,sdb,sdc,sdd
----total-cpu-usage---- -dsk/total----dsk/sda-----dsk/sdb-----dsk/sdc-----dsk/sdd-- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ: read  writ: read  writ: read  writ: read  writ| recv  send|  in   out | int   csw 
 15   3  20  61   0   1|3118k 3957k:3112k  654k:6684B 3303k: 105B   79B:  19B   79B|   0     0 |3701B 4301B| 601  1596 
  6   1   7  84   1   1| 968k 1920k: 960k  672k:8192B 1248k:   0     0 :   0     0 |3191B   39k|   0     0 | 494  2617 
 13   3   6  77   0   1| 912k  920k: 896k  204k:  16k  716k:   0     0 :   0     0 |6183B   64k|   0     0 | 491  2130 
 22   2   7  68   0   0| 880k 1504k: 872k  496k:8192B 1008k:   0     0 :   0     0 |  24k   80k|   0     0 | 603  2157 
  1   1   6  92   0   0| 996k 1640k: 992k  500k:4096B 1140k:   0     0 :   0     0 |2509B   52k|   0     0 | 476  2133 
  6   2   5  87   0   0| 952k 2720k: 940k 1072k:  12k 1648k:   0     0 :   0     0 |2782B   49k|   0     0 | 465  1977 
 20   1   5  74   0   0| 880k  968k: 872k  228k:8192B  740k:   0     0 :   0     0 |3684B   30k|   0     0 | 449  1860 
 10   2   8  79   0   1| 920k 2008k: 916k  716k:4096B 1292k:   0     0 :   0     0 |1874B   41k|   0     0 | 467  2081 
  6   1   7  86   0   0| 864k 2160k: 848k  824k:  16k 1336k:   0     0 :   0     0 |2193B   30k|   0     0 | 433  1840 
  2   0   3  91   2   2| 316k 2872k: 308k 1064k:8192B 1808k:   0     0 :   0     0 |  18k   36k|   0     0 | 747  2234 
  2   2   0  94   0   2|   0  3060k:   0  1664k:   0  1396k:   0     0 :   0     0 |  92k   45k|   0     0 | 857  4393 
  7   1   3  88   0   1| 680k 2540k: 676k 1188k:4096B 1352k:   0     0 :   0     0 |5998B   45k|   0     0 | 593  3203 



Jeff.