[TriLUG] help with fuser/ssh reporting lots of processes
Tim Jowers via TriLUG
trilug at trilug.org
Mon Jul 18 09:18:26 EDT 2016
Thank you Matt,
Those are GREAT commands. Right now, everything looks good with those. I
only recognize my activity in them. I see some connection attempts from
places, including what is supposedly a network security company "
protected.javapipe.com" but nobody getting in any more. So, I think the
Chinese illegally accessed the system by cracking the root password and
because I had naively left root access open on ssh, thinking an 18
character, complex password would be sufficient.
Thanks, back to $JOB.
Tim
On Mon, Jul 18, 2016 at 9:00 AM, Matt Flyer via TriLUG <trilug at trilug.org>
wrote:
> Here is the list of commands that I would recommend running to try to
> cross correlate the open connections via a process:
>
> (run each individually and save the output) ps acxfwwwe, lsof -Pwln,
> and netstat -anpe
>
> You should also scour your log files (consider running them through
> logwatch.
>
> Ultimately, I think you will want to rebuild the system image, but I
> also think it is vitally important to try to identify how they got in
> so that you can hopefully defend against it going forward.
>
> It is kind of hard to tell from the LSOF output, but it looks like they
> may have launched copies of SSHD as root, which would mean a root level
> compromise.
>
> Places like /tmp, which are relatively insecure are common locations
> where you can find malware binaries.
>
> You could also try to run a chesksum (md5 or sha) of your system
> binaries versus the repository to see if any of the system files, e.g.
> ssh, have been replaced.
>
> Apache or other web servers are another common intrusion tactic,
> especially if they can be made to do a remote download (remote file
> inclusion I think it is called).
>
> On Mon, 2016-07-18 at 08:25 -0400, William Sutton via TriLUG wrote:
> > anything in /var/log/secure?
> >
> > William Sutton
> >
> > On Mon, 18 Jul 2016, Tim Jowers via TriLUG wrote:
> >
> > >
> > > Hi,
> > >
> > > I run these two less than a second apart:
> > >
> > > [root at test1 log]# fuser ssh/tcp
> > >
> > > ssh/tcp: 685 5066 5283 5284 5289 5290 5291 529
> > > 2 5293
> > > 5294
> > >
> > > [root at test1 log]# fuser ssh/tcp
> > >
> > > ssh/tcp: 685 5066 5289 5290 5293 5294 5296 529
> > > 7 5298
> > > 5299
> > >
> > >
> > > Any ideas how to troubleshoot? I think I have some Chinese
> > > search bot
> > > malware based on this:
> > >
> > > [root at test1 log]# lsof -i
> > >
> > > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> > >
> > > sshd 685 root 3u IPv6 350221175 0t0 TCP *:ssh
> > > (LISTEN)
> > >
> > > sshd 685 root 4u IPv4 350221177 0t0 TCP *:ssh
> > > (LISTEN)
> > >
> > > mysqld 811 mysql 10u IPv4 350221673 0t0 TCP *:mysql
> > > (LISTEN)
> > >
> > > sshd 5066 root 3r IPv4 4054471422 0t0 TCP
> > > 198-20-184-56-host.colocrossing.com:ssh->
> > > cpe-45-37-198-154.nc.res.rr.com:59006 (ESTABLISHED)
> > >
> > > sshd 5361 root 3r IPv4 4054875967 0t0 TCP
> > > 198-20-184-57-host.colocrossing.com:ssh->112.85.42.99:41796
> > > (ESTABLISHED)
> > >
> > > sshd 5362 sshd 3u IPv4 4054875967 0t0 TCP
> > > 198-20-184-57-host.colocrossing.com:ssh->112.85.42.99:41796
> > > (ESTABLISHED)
> > >
> > > sshd 5365 root 3r IPv4 4054877149 0t0 TCP
> > > 198-20-184-56-host.colocrossing.com:ssh->112.85.42.99:openmailpxy
> > > (ESTABLISHED)
> > >
> > > sshd 5366 sshd 3u IPv4 4054877149 0t0 TCP
> > > 198-20-184-56-host.colocrossing.com:ssh->112.85.42.99:openmailpxy
> > > (ESTABLISHED)
> > >
> > > sshd 5369 root 3r IPv4 4054886185 0t0 TCP
> > > 198-20-184-56-host.colocrossing.com:ssh->221.229.172.99:36122
> > > (ESTABLISHED)
> > >
> > > sshd 5370 sshd 3u IPv4 4054886185 0t0 TCP
> > > 198-20-184-56-host.colocrossing.com:ssh->221.229.172.99:36122
> > > (ESTABLISHED)
> > >
> > > sshd 5371 root 3r IPv4 4054886747 0t0 TCP
> > > 198-20-184-57-host.colocrossing.com:ssh->221.229.172.99:15096
> > > (ESTABLISHED)
> > >
> > > sshd 5372 sshd 3u IPv4 4054886747 0t0 TCP
> > > 198-20-184-57-host.colocrossing.com:ssh->221.229.172.99:15096
> > > (ESTABLISHED)
> > >
> > > java 18216 root 43u IPv6 3405192816 0t0 TCP
> > > *:webcache
> > > (LISTEN)
> > >
> > > java 18216 root 48u IPv6 3405192820 0t0 TCP *:8009
> > > (LISTEN)
> > >
> > > java 18216 root 72u IPv6 3405192937 0t0 TCP
> > > localhost.localdomain:8005 (LISTEN)
> > >
> > > httpd 26003 apache 3u IPv6 3253453758 0t0 TCP *:http
> > > (LISTEN)
> > >
> > > httpd 26361 apache 3u IPv6 3253453758 0t0 TCP *:http
> > > (LISTEN)
> > >
> > > httpd 27165 apache 3u IPv6 3253453758 0t0 TCP *:http
> > > (LISTEN)
> > >
> > > httpd 27818 root 3u IPv6 3253453758 0t0 TCP *:http
> > > (LISTEN)
> > >
> > > and
> > >
> > > [root at test1 log]# netstat -a
> > >
> > > Active Internet connections (servers and established)
> > >
> > > Proto Recv-Q Send-Q Local Address Foreign Address
> > > State
> > >
> > > tcp 0 0 *:ssh *:*
> > > LISTEN
> > >
> > > tcp 0 0 *:mysql *:*
> > > LISTEN
> > >
> > > tcp 0 0 198-20-184-57-host.colo:ssh 112.85.42.99:15265
> > > ESTABLISHED
> > >
> > > tcp 0 0 198-20-184-56-host.colo:ssh
> > > 221.229.172.99:48079
> > > TIME_WAIT
> > >
> > > tcp 0 0 198-20-184-56-host.colo:ssh
> > > 221.229.172.99:33195
> > > ESTABLISHED
> > >
> > > tcp 0 0 198-20-184-57-host.colo:ssh
> > > 221.229.172.99:44556
> > > ESTABLISHED
> > >
> > > tcp 0 0 198-20-184-57-host.colo:ssh
> > > 221.229.172.99:15096
> > > TIME_WAIT
> > >
> > > tcp 0 608 198-20-184-56-host.colo:ssh cpe-45-37-198-
> > > 154.nc.:59006
> > > ESTABLISHED
> > >
> > > tcp 0 0 198-20-184-56-host.colo:ssh 112.85.42.99:42180
> > > ESTABLISHED
> > >
> > > tcp 0 0 *:webcache *:*
> > > LISTEN
> > >
> > > tcp 0 0 *:http *:*
> > > LISTEN
> > >
> > > tcp 0 0 *:ssh *:*
> > > LISTEN
> > >
> > > tcp 0 0 localhost.localdomain:8005 *:*
> > > LISTEN
> > >
> > > tcp 0 0 *:8009 *:*
> > > LISTEN
> > >
> > > tcp 0 0 198-20-184-56-host.col:http ns336619.ip-37-187-
> > > 16:18286
> > > TIME_WAIT
> > >
> > > tcp 0 0 198-20-184-56-host.col:http
> > > hydrogen081.a.ahrefs.:30831
> > > TIME_WAIT
> > >
> > > and some StackOverflow article where someone posted that
> > > *221.229.172.99*
> > > is a Chinese search botnet.
> > >
> > > last and lastlog don't show anything. There is no /var/log/auth.log
> > > present. Not sure if there should be. Just tried things based on
> > > Internet
> > > searching.
> > >
> > > I guess there is no easy way to kill this? Sounds like I should
> > > just ask
> > > for a new server instance (ChicagoVPS)? I use SVN to back up my
> > > files there.
> > >
> > >
> > > Thanks for any ideas.
> > >
> > > Tim
> > > --
> > > This message was sent to: William <william at trilug.org>
> > > To unsubscribe, send a blank message to trilug-leave at trilug.org
> > > from that address.
> > > TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> > > Unsubscribe or edit options on the web : http://www.trilug.o
> > > rg/mailman/options/trilug/william%40trilug.org
> > > Welcome to TriLUG: http://trilug.org/welcome
> --
> This message was sent to: timjowers <timjowers at gmail.com>
> To unsubscribe, send a blank message to trilug-leave at trilug.org from that
> address.
> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web :
> http://www.trilug.org/mailman/options/trilug/timjowers%40gmail.com
> Welcome to TriLUG: http://trilug.org/welcome
>
More information about the TriLUG
mailing list