[TriLUG] help with fuser/ssh reporting lots of processes
Matt Flyer via TriLUG
trilug at trilug.org
Mon Jul 18 09:00:20 EDT 2016
Here is the list of commands that I would recommend running to try to
cross correlate the open connections via a process:
(run each individually and save the output) ps acxfwwwe, lsof -Pwln,
and netstat -anpe
You should also scour your log files (consider running them through
logwatch.
Ultimately, I think you will want to rebuild the system image, but I
also think it is vitally important to try to identify how they got in
so that you can hopefully defend against it going forward.
It is kind of hard to tell from the LSOF output, but it looks like they
may have launched copies of SSHD as root, which would mean a root level
compromise.
Places like /tmp, which are relatively insecure are common locations
where you can find malware binaries.
You could also try to run a chesksum (md5 or sha) of your system
binaries versus the repository to see if any of the system files, e.g.
ssh, have been replaced.
Apache or other web servers are another common intrusion tactic,
especially if they can be made to do a remote download (remote file
inclusion I think it is called).
On Mon, 2016-07-18 at 08:25 -0400, William Sutton via TriLUG wrote:
> anything in /var/log/secure?
>
> William Sutton
>
> On Mon, 18 Jul 2016, Tim Jowers via TriLUG wrote:
>
> >
> > Hi,
> >
> > I run these two less than a second apart:
> >
> > [root at test1 log]# fuser ssh/tcp
> >
> > ssh/tcp: 685 5066 5283 5284 5289 5290 5291 529
> > 2 5293
> > 5294
> >
> > [root at test1 log]# fuser ssh/tcp
> >
> > ssh/tcp: 685 5066 5289 5290 5293 5294 5296 529
> > 7 5298
> > 5299
> >
> >
> > Any ideas how to troubleshoot? I think I have some Chinese
> > search bot
> > malware based on this:
> >
> > [root at test1 log]# lsof -i
> >
> > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> >
> > sshd 685 root 3u IPv6 350221175 0t0 TCP *:ssh
> > (LISTEN)
> >
> > sshd 685 root 4u IPv4 350221177 0t0 TCP *:ssh
> > (LISTEN)
> >
> > mysqld 811 mysql 10u IPv4 350221673 0t0 TCP *:mysql
> > (LISTEN)
> >
> > sshd 5066 root 3r IPv4 4054471422 0t0 TCP
> > 198-20-184-56-host.colocrossing.com:ssh->
> > cpe-45-37-198-154.nc.res.rr.com:59006 (ESTABLISHED)
> >
> > sshd 5361 root 3r IPv4 4054875967 0t0 TCP
> > 198-20-184-57-host.colocrossing.com:ssh->112.85.42.99:41796
> > (ESTABLISHED)
> >
> > sshd 5362 sshd 3u IPv4 4054875967 0t0 TCP
> > 198-20-184-57-host.colocrossing.com:ssh->112.85.42.99:41796
> > (ESTABLISHED)
> >
> > sshd 5365 root 3r IPv4 4054877149 0t0 TCP
> > 198-20-184-56-host.colocrossing.com:ssh->112.85.42.99:openmailpxy
> > (ESTABLISHED)
> >
> > sshd 5366 sshd 3u IPv4 4054877149 0t0 TCP
> > 198-20-184-56-host.colocrossing.com:ssh->112.85.42.99:openmailpxy
> > (ESTABLISHED)
> >
> > sshd 5369 root 3r IPv4 4054886185 0t0 TCP
> > 198-20-184-56-host.colocrossing.com:ssh->221.229.172.99:36122
> > (ESTABLISHED)
> >
> > sshd 5370 sshd 3u IPv4 4054886185 0t0 TCP
> > 198-20-184-56-host.colocrossing.com:ssh->221.229.172.99:36122
> > (ESTABLISHED)
> >
> > sshd 5371 root 3r IPv4 4054886747 0t0 TCP
> > 198-20-184-57-host.colocrossing.com:ssh->221.229.172.99:15096
> > (ESTABLISHED)
> >
> > sshd 5372 sshd 3u IPv4 4054886747 0t0 TCP
> > 198-20-184-57-host.colocrossing.com:ssh->221.229.172.99:15096
> > (ESTABLISHED)
> >
> > java 18216 root 43u IPv6 3405192816 0t0 TCP
> > *:webcache
> > (LISTEN)
> >
> > java 18216 root 48u IPv6 3405192820 0t0 TCP *:8009
> > (LISTEN)
> >
> > java 18216 root 72u IPv6 3405192937 0t0 TCP
> > localhost.localdomain:8005 (LISTEN)
> >
> > httpd 26003 apache 3u IPv6 3253453758 0t0 TCP *:http
> > (LISTEN)
> >
> > httpd 26361 apache 3u IPv6 3253453758 0t0 TCP *:http
> > (LISTEN)
> >
> > httpd 27165 apache 3u IPv6 3253453758 0t0 TCP *:http
> > (LISTEN)
> >
> > httpd 27818 root 3u IPv6 3253453758 0t0 TCP *:http
> > (LISTEN)
> >
> > and
> >
> > [root at test1 log]# netstat -a
> >
> > Active Internet connections (servers and established)
> >
> > Proto Recv-Q Send-Q Local Address Foreign Address
> > State
> >
> > tcp 0 0 *:ssh *:*
> > LISTEN
> >
> > tcp 0 0 *:mysql *:*
> > LISTEN
> >
> > tcp 0 0 198-20-184-57-host.colo:ssh 112.85.42.99:15265
> > ESTABLISHED
> >
> > tcp 0 0 198-20-184-56-host.colo:ssh
> > 221.229.172.99:48079
> > TIME_WAIT
> >
> > tcp 0 0 198-20-184-56-host.colo:ssh
> > 221.229.172.99:33195
> > ESTABLISHED
> >
> > tcp 0 0 198-20-184-57-host.colo:ssh
> > 221.229.172.99:44556
> > ESTABLISHED
> >
> > tcp 0 0 198-20-184-57-host.colo:ssh
> > 221.229.172.99:15096
> > TIME_WAIT
> >
> > tcp 0 608 198-20-184-56-host.colo:ssh cpe-45-37-198-
> > 154.nc.:59006
> > ESTABLISHED
> >
> > tcp 0 0 198-20-184-56-host.colo:ssh 112.85.42.99:42180
> > ESTABLISHED
> >
> > tcp 0 0 *:webcache *:*
> > LISTEN
> >
> > tcp 0 0 *:http *:*
> > LISTEN
> >
> > tcp 0 0 *:ssh *:*
> > LISTEN
> >
> > tcp 0 0 localhost.localdomain:8005 *:*
> > LISTEN
> >
> > tcp 0 0 *:8009 *:*
> > LISTEN
> >
> > tcp 0 0 198-20-184-56-host.col:http ns336619.ip-37-187-
> > 16:18286
> > TIME_WAIT
> >
> > tcp 0 0 198-20-184-56-host.col:http
> > hydrogen081.a.ahrefs.:30831
> > TIME_WAIT
> >
> > and some StackOverflow article where someone posted that
> > *221.229.172.99*
> > is a Chinese search botnet.
> >
> > last and lastlog don't show anything. There is no /var/log/auth.log
> > present. Not sure if there should be. Just tried things based on
> > Internet
> > searching.
> >
> > I guess there is no easy way to kill this? Sounds like I should
> > just ask
> > for a new server instance (ChicagoVPS)? I use SVN to back up my
> > files there.
> >
> >
> > Thanks for any ideas.
> >
> > Tim
> > --
> > This message was sent to: William <william at trilug.org>
> > To unsubscribe, send a blank message to trilug-leave at trilug.org
> > from that address.
> > TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> > Unsubscribe or edit options on the web : http://www.trilug.o
> > rg/mailman/options/trilug/william%40trilug.org
> > Welcome to TriLUG: http://trilug.org/welcome
More information about the TriLUG
mailing list