[TriLUG] TriLUG.org is up. Logins are working.

[Matt Frye]

You can now login to login.trilug.org and the website is back up.  At
this point, the root cause appears to be the failure to allocate
memory that occured at 09:10:38...

  Mar 17 09:10:38 talon kernel: __alloc_pages: 0-order allocation
failed (gfp=0x1d2/0)

Why we ran out of memory is unclear, but some symptoms were that:

1) the kerberos kdc went down (talon)
2) slapd went down (talon)
3) named went down (talon)
4) postfix went down (talon)
5) apache2 went down (talon)
6) sshd went down (dargo)
7) other lesser daemons started choking due to the high load after
postfix, etc were restarted.

Additionally, attempts to revive apache2 resulted in the following error:

 [crit] (28)No space left on device: mod_rewrite: could not create
rewrite_log_lock

which was the system running out of space to store semaphores.  This
problem was quickly remedied by removing semaphore arrays.

Your TriLUG Sysadmin Team