[TriLUG] System overload issues

Brian McCullough bdmc at buadh-brath.com
Fri May 24 10:50:39 EDT 2013


On Fri, May 24, 2013 at 07:35:53AM -0700, John Vaughters wrote:
> Brian,
> 
> I highly recommend reading up on sysstat and sar command. This gives you quick system stats that will point you to many problems and when they occur. With that info you can narrow in on your other logs for further information. You can find out about CPU, MEM, SWAP, I/O waits (disk), NIC. Centos usually defaults to 7 days of stats, but yo can increase that if you like.

Yes, I have used sar when I work on AIX machines, and have also tried it
on this machine, back when I first starting trying to diagnose this
issue. ( these issues? )



> My favorite sar command to view yesterday's stats:
> 
> sar -f /var/log/sa/sa$(date +%d -d yesterday)

OK.  Part of the frustration is that when I do so, things LOOK all
right.  For instance:


# sar -f /var/log/sa/sa$(date +%d -d yesterday)
Linux 2.6.18-238.19.1.el5 (hostname) 	05/23/2013

12:00:06 AM       CPU     %user     %nice   %system   %iowait    %steal %idle
12:10:05 AM       all     55.26      0.00     44.71      0.00      0.00 0.04
12:20:01 AM       all     55.73      0.00     44.09      0.06      0.00 0.12
12:30:02 AM       all     57.24      0.00     42.32      0.07      0.00 0.38
12:40:30 AM       all     48.63      0.00     40.60      6.45      0.00 4.32
01:10:12 AM       all     27.86      0.00     69.94      1.58      0.00 0.62
01:20:05 AM       all     55.80      0.00     44.08      0.02      0.00 0.10
01:30:04 AM       all     55.76      0.00     44.19      0.00      0.00 0.05
01:40:02 AM       all     55.79      0.00     44.17      0.00      0.00 0.04
01:50:01 AM       all     41.59      0.00     31.48      3.79      0.00 23.14
02:00:02 AM       all     39.43      0.00     28.29     10.35      0.00 21.93
02:10:02 AM       all     54.48      0.00     45.47      0.01      0.00 0.04
02:30:26 AM       all     26.78      0.00     25.28     47.72      0.00 0.22
02:40:24 AM       all     16.37      0.00     11.03     71.29      0.00 1.31
02:50:16 AM       all     16.46      0.00     10.94     72.60      0.00 0.00
03:02:20 AM       all     28.06      0.00     19.90     51.99      0.00 0.04
03:10:28 AM       all     52.36      0.00     39.77      7.87      0.00 0.00
03:20:15 AM       all     56.13      0.00     43.83      0.01      0.00 0.03
03:30:06 AM       all     56.08      0.00     43.88      0.00      0.00 0.04
03:40:02 AM       all     56.04      0.00     43.93      0.00      0.00 0.03
03:50:10 AM       all     56.13      0.00     43.83      0.00      0.00 0.03
04:00:48 AM       all     55.05      0.00     42.90      1.72      0.00 0.33
04:10:03 AM       all     13.40      0.07      7.93     74.43      0.00 4.17
04:20:10 AM       all     44.25      0.02     38.17      9.07      0.00 8.50
05:01:08 AM       all      0.76      0.00     87.15     11.59      0.00 0.50
05:10:49 AM       all      1.02      0.00     80.80     17.42      0.00 0.76
05:20:47 AM       all      0.92      0.00     77.51     20.34      0.00 1.23
05:41:02 AM       all      0.62      0.00     81.72     16.85      0.00 0.81
05:51:17 AM       all      0.52      0.00     83.59     15.23      0.00 0.67
06:00:43 AM       all      0.60      0.00     76.22     21.63      0.00 1.54
06:10:22 AM       all      0.85      0.01     68.45     27.38      0.00 3.31
06:20:05 AM       all      2.05      0.01     44.17     32.96      0.00 20.83
06:30:18 AM       all      0.20      0.00      1.20     45.94      0.00 52.66
06:41:15 AM       all      2.76      0.02      3.88     42.90      0.00 50.45
06:50:11 AM       all      5.33      0.00      4.45     66.51      0.00 23.70
07:00:32 AM       all     16.85      0.00     13.94     63.53      0.00 5.67
07:10:25 AM       all     27.87      0.01     21.85     48.47      0.00 1.80
07:20:02 AM       all     50.65      0.04     41.06      7.03      0.00 1.22
07:30:16 AM       all     32.75      0.02     24.72     42.49      0.00 0.03
07:40:05 AM       all     55.82      0.01     44.13      0.01      0.00 0.03
07:53:59 AM       all     14.12      0.01     12.92     72.34      0.00 0.61
08:01:57 AM       all     10.90      0.00      9.14     79.76      0.00 0.19
08:10:07 AM       all     10.60      0.01      8.74     80.10      0.00 0.55

08:10:07 AM       CPU     %user     %nice   %system   %iowait    %steal %idle
08:20:01 AM       all     27.38      0.05     20.27     43.95      0.00 8.35
08:30:01 AM       all     44.07      0.11     27.82      3.49      0.00 24.51
08:40:01 AM       all     18.56      0.30      4.03      6.41      0.00 70.70
08:50:01 AM       all     12.77      0.28      4.43      2.72      0.00 79.79
09:00:01 AM       all     12.56      0.15      4.14      2.50      0.00 80.65
09:10:01 AM       all     13.37      0.08      4.83      1.36      0.00 80.36
09:20:01 AM       all     14.44      0.05      4.96      1.34      0.00 79.21
09:30:01 AM       all     12.91      0.07      4.24      1.11      0.00 81.68
09:40:01 AM       all     12.35      0.35      4.08      2.16      0.00 81.06
09:50:01 AM       all     13.38      0.04      4.60      0.84      0.00 81.15
10:00:01 AM       all     13.55      0.04      4.47      1.02      0.00 80.92
10:10:01 AM       all     14.18      0.04      4.57      1.07      0.00 80.13
10:20:01 AM       all     14.59      0.05      4.93      1.07      0.00 79.36
10:30:02 AM       all     16.64      0.04      5.95      1.01      0.00 76.36
10:40:01 AM       all     14.47      0.27      5.20      1.07      0.00 78.98
10:50:01 AM       all     13.61      0.24      4.61      1.40      0.00 80.13
11:00:01 AM       all     17.15      0.27      6.23      2.14      0.00 74.22
11:10:01 AM       all     17.21      0.25      6.07      2.21      0.00 74.26
11:20:01 AM       all     16.97      0.23      6.06      1.96      0.00 74.79
11:30:01 AM       all     18.95      0.21      6.65      1.75      0.00 72.44
11:40:01 AM       all     19.58      0.22      7.56      2.06      0.00 70.58
11:50:01 AM       all     19.09      0.22      7.71      1.77      0.00 71.21
12:00:01 PM       all     18.57      0.26      6.84      2.48      0.00 71.84
12:10:01 PM       all     16.12      0.27      5.24      1.93      0.00 76.43
12:20:01 PM       all     16.12      0.26      5.27      1.69      0.00 76.66
12:30:01 PM       all     24.32      0.17     11.95      0.99      0.00 62.57
12:40:01 PM       all     27.39      0.14     16.14      0.88      0.00 55.44
12:50:01 PM       all     16.86      0.21      5.70      1.96      0.00 75.27
01:00:01 PM       all     16.04      0.23      5.62      1.75      0.00 76.36
01:10:01 PM       all     16.41      0.24      5.19      1.75      0.00 76.40
01:20:01 PM       all     15.40      0.28      4.84      1.96      0.00 77.52
01:30:01 PM       all     17.04      0.24      5.61      1.58      0.00 75.55
01:40:02 PM       all     18.67      0.23      6.05      1.65      0.00 73.40
01:50:01 PM       all     15.38      0.23      5.03      1.63      0.00 77.73
02:00:01 PM       all     14.66      0.21      4.87      1.99      0.00 78.27
02:10:01 PM       all     15.56      0.16      4.87      1.78      0.00 77.63
02:20:01 PM       all     15.60      0.23      4.93      1.54      0.00 77.70
02:30:01 PM       all     17.98      0.24      6.28      1.54      0.00 73.97
02:40:01 PM       all     18.10      0.18      6.30      1.14      0.00 74.28
02:50:01 PM       all     14.98      0.26      5.26      1.84      0.00 77.67
03:00:01 PM       all     13.61      0.24      4.65      1.00      0.00 80.50
03:10:01 PM       all     13.93      0.20      4.59      0.94      0.00 80.34

03:10:01 PM       CPU     %user     %nice   %system   %iowait    %steal %idle
03:20:01 PM       all     14.10      0.21      4.99      0.84      0.00 79.85
03:30:01 PM       all     14.45      0.22      4.95      0.67      0.00 79.71
03:40:01 PM       all     14.91      0.21      5.29      0.86      0.00 78.73
03:50:01 PM       all     14.12      0.23      4.93      0.84      0.00 79.88
04:00:01 PM       all      8.31      0.02      3.47      0.83      0.00 87.37
04:10:01 PM       all      4.06      0.00      1.38      0.60      0.00 93.96
04:20:01 PM       all      5.50      0.00      2.00      0.57      0.00 91.92
04:30:01 PM       all      7.49      0.00      2.65      0.58      0.00 89.28
04:40:01 PM       all     10.55      0.00      3.95      0.60      0.00 84.90
04:50:01 PM       all      3.65      0.00      1.10      0.54      0.00 94.71
05:00:01 PM       all      4.76      0.00      1.71      0.56      0.00 92.97
05:10:01 PM       all      8.07      0.00      2.79      0.82      0.00 88.32
05:20:01 PM       all      6.52      0.00      2.51      0.63      0.00 90.34
05:30:01 PM       all      4.30      0.00      1.32      0.50      0.00 93.89
05:40:01 PM       all     11.39      0.00      5.04      0.41      0.00 83.16
05:50:01 PM       all      5.57      0.00      1.74      0.60      0.00 92.08
06:00:01 PM       all      7.28      0.00      2.20      0.62      0.00 89.89
06:10:01 PM       all      3.36      0.00      1.00      0.59      0.00 95.04
06:20:01 PM       all      4.01      0.00      1.25      0.60      0.00 94.13
06:30:02 PM       all      4.43      0.00      1.39      0.48      0.00 93.69
06:40:02 PM       all      5.53      0.00      1.82      0.54      0.00 92.12
06:50:01 PM       all      4.16      0.00      1.59      0.55      0.00 93.70
07:00:01 PM       all      3.17      0.00      1.03      0.47      0.00 95.33
07:10:01 PM       all      3.92      0.00      1.39      0.70      0.00 93.99
07:20:01 PM       all      3.31      0.00      0.98      0.55      0.00 95.15
07:30:01 PM       all      3.89      0.00      1.25      0.55      0.00 94.31
07:40:01 PM       all      7.95      0.00      2.84      0.60      0.00 88.60
07:50:01 PM       all      6.17      0.00      2.23      0.47      0.00 91.13
08:00:01 PM       all      3.96      0.00      1.18      0.61      0.00 94.25
08:10:01 PM       all      5.08      0.00      1.60      0.74      0.00 92.56
08:20:01 PM       all      4.05      0.00      1.19      0.62      0.00 94.13
08:30:01 PM       all      4.31      0.00      1.52      0.62      0.00 93.54
08:40:01 PM       all     13.90      0.00      9.08      0.46      0.00 76.56
08:50:01 PM       all     36.79      0.00     29.68      0.24      0.00 33.29
09:00:01 PM       all      4.05      0.00      1.33      0.80      0.00 93.83
09:10:01 PM       all      5.46      0.00      1.84      0.77      0.00 91.94
09:20:01 PM       all      4.31      0.00      1.64      0.64      0.00 93.40
09:30:01 PM       all      3.64      0.00      1.26      0.81      0.00 94.29
09:40:01 PM       all      5.50      0.00      2.13      0.72      0.00 91.64
09:50:01 PM       all      7.55      0.00      2.73      0.74      0.00 88.98
10:00:01 PM       all      4.55      0.00      1.63      0.90      0.00 92.91
10:10:01 PM       all      4.14      0.00      1.45      0.64      0.00 93.77

10:10:01 PM       CPU     %user     %nice   %system   %iowait    %steal %idle
10:20:01 PM       all      5.71      0.00      2.01      0.51      0.00 91.78
10:30:01 PM       all      3.50      0.00      1.23      0.56      0.00 94.70
10:40:01 PM       all      3.58      0.00      1.27      0.50      0.00 94.65
10:50:01 PM       all      4.66      0.00      1.72      0.58      0.00 93.04
11:00:01 PM       all      4.09      0.00      1.45      0.46      0.00 94.00
11:10:01 PM       all      5.72      0.00      1.88      0.45      0.00 91.95
11:20:01 PM       all      2.79      0.00      1.01      0.42      0.00 95.78
11:30:01 PM       all      3.66      0.00      1.34      0.34      0.00 94.67
11:40:01 PM       all      3.66      0.00      1.53      0.34      0.00 94.47
11:50:01 PM       all      4.37      0.00      1.53      0.41      0.00 93.69
Average:          all     16.83      0.06     17.93      9.24      0.00 55.93



This is the day where the web sites were basically non-responsive from
some time between 1am and 2am until about 8:30am, and I do know that
both "w" and "top" were showing system loads of 200 or more.


Brian




More information about the TriLUG mailing list