[TriLUG] System overload issues

Ron Kelley rkelleyrtp at gmail.com
Fri May 24 11:46:06 EDT 2013


(First reply is held up by the moderators - message too long)...

Brian,

From the sar output, the system is really getting hammered between midnight and 8:30am (extremely high I/O wait and high CPU).  I suspect you are running out of RAM and are swapping like crazy during this time.  From 9am until 6PM (normal business hours), the box seems to be handling the load very well with little I/O wait and relatively hight %idle numbers.  It is strange your issues are during the daytime when the system is mostly idle instead of at night...

Just going off the sar output, I suggest (a) putting in more RAM to minimize the swapping, or (b) putting an SSD in that box to increase your IO response times.  My suspicion is you are running slower 7200RPM SATA drives that just cannot keep up with the I/O workload.   A 512GB Samsung 840 series SSD costs about $500 (NewEgg) right now, and would certainly fit the bill toward increasing your system's response times.  

My $0.02

-Ron



Thanks,


-Ron

On May 24, 2013, at 10:50 AM, Brian McCullough wrote:

> On Fri, May 24, 2013 at 07:35:53AM -0700, John Vaughters wrote:
>> Brian,
>> 
>> I highly recommend reading up on sysstat and sar command. This gives you quick system stats that will point you to many problems and when they occur. With that info you can narrow in on your other logs for further information. You can find out about CPU, MEM, SWAP, I/O waits (disk), NIC. Centos usually defaults to 7 days of stats, but yo can increase that if you like.
> 
> Yes, I have used sar when I work on AIX machines, and have also tried it
> on this machine, back when I first starting trying to diagnose this
> issue. ( these issues? )
> 
> 
> 
>> My favorite sar command to view yesterday's stats:
>> 
>> sar -f /var/log/sa/sa$(date +%d -d yesterday)
> 
> OK.  Part of the frustration is that when I do so, things LOOK all
> right.  For instance:
> 




More information about the TriLUG mailing list