[TriLUG] Question - Linux Server CPR

[Scott Chilcote]

Hi Folks,

I had someone call me yesterday who had a seriously ill Linux server.

He said that the machine seemed heavily overloaded and was running very 
slowly.  It would respond to its keyboard at the console, but only at 
the rate of one keystroke every five to ten minutes.  Attempts to reach 
it with ssh from another system were timing out with "connection reset 
by peer".  control-alt-delete wasn't having any affect (the command may 
have been removed for security).

He wound up hitting the reset button, and now has hard disk errors that 
e3fsck can't handle.  Quite a mess.  It's possible that he could have 
logged in and issued commands (if he'd kept at it all night), but the 
slowdown appeared to be worsening.

What he hoped I could tell him was whether there was any way to get the 
kernel's attention under these circumstances.  The problem might have 
been fixed by deleting an oversize /tmp file or killing some processes, 
if there was a way to get in.  I don't know of any.

Is there a better way to handle this problem, other than "Don't let it 
happen to begin with"?

Thanks,

Scott C.