[TriLUG] Help diagnosing a hijacked mail server?

[Matt Flyer]

Here are some things to get your compromise investigation started:
1) review the CERT intruder detection checklist.  It outlines the steps
and has the commands to find hidden and modified files:
http://web.archive.org/web/20080109214340/http://www.cert.org/tech_tips/intruder_detection_checklist.html
2) What distribution are you running, what revision level? What server
and content manager applications are you running (e.g. Apache, SSH, FTP,
atmail, CPanal, plex, nagios) and what revision level. What would you
say the overall patch state of the system is? Do you routinely apply
updates? Do you have any sort of intrusion prevent software in place,
e.g. Aide or Ossec? Do you use a firewall to close unused ports and are
you running SELinux?
3) Capture a snapshot of the process tree, network connections, and open
files. Be sure to run this command as root, else it will miss important
process information that will help to link the pieces together:
( /bin/ps axfwwwe -opid,ppid,uid,context,cmd 2>&1; /usr/sbin/lsof -Pwln
2>&1; \find /var/spool/cron 2>&1; /bin/netstat -anpe
2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last \-wai 2>&1; /usr/bin/who -a
2>&1 ) > /tmp/logfile 2>&1
4) Perform a detailed analysis of your log files. The best way to get
started with this is to use the tool logwatch.  This is why I suggested
copying the files.  Again, run it as root or else you won't be able to
access all of the log files (this works on Centos, for debian you may
need to use the --save /path to log instead of > redirect): 
logwatch --detail High --service All --range All --archives --numeric
> /path/to/logwatch.log
5) Using the Cert checklist, perform the steps for finding hidden files and scripts (step 9). Pay particular attention to /dev, /temp/, and /proc. Also look for any files with setuid or setgid (step 2), check all of the cron and at tables for modifications (be sure to look in /var/spool/cron).
6)Try to verify your system binaries. If you are using Centos or another RPM based distrubtion, use RPM -vV. If you are using a debian based system, use debsums.

The command in step 3 will capture a lot of information and place it in
a file called logfile in your temp directory.  Mostly this will The
information collected will need to be analyzed very carefully.  It will
contain a cross linked process tree output, which should show the
process sending the mail and hopefully a chain that shows where it is
originating.  Your log files will hopefully contain information
pertaining to the initial event showing how it occurred.

The good thing about email intrusions like this is that they don't
necessarily require root level privilege to run. 

If you have any questions, please reply back.

On Tue, 2012-10-09 at 03:45 -0500, Phillip Rhodes wrote:

> Hey guys, hoping someone can help me out a bit here.  I have a postfix
> server running, serving one of my domains, and all of the "open relay
> test" things tell me it's not relaying mail openly (and it shouldn't
> be, 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://www.trilug.org/pipermail/trilug/attachments/20121009/a35f3813/attachment.pgp>