[TriLUG] help with fuser/ssh reporting lots of processes
Roger via TriLUG
trilug at trilug.org
Mon Jul 18 16:54:22 EDT 2016
If you believe you've been hacked, re-install Linux. No ifs, ands, or buts. No way around it. Be safe. Reformat thumb drives. Etc.
I saw a reference to a data base in one of your messages. One standard probe for hacking is to send tests to try and gain entry via a database, taking advantage of vulnerabilities therein. Google
hack mysql vulnerability
Looking for traces of like probing pointed back to Chinese sources on a regular basis while working in the past screening for attempts to break in. 10k probes per week, mostly from China.
--
Roger Broseus
(Please excuse tiepos induzed by the droided predictive text tool.)
On July 18, 2016 9:18:26 AM EDT, Tim Jowers via TriLUG <trilug at trilug.org> wrote:
>Thank you Matt,
>
>Those are GREAT commands. Right now, everything looks good with those.
>I
>only recognize my activity in them. I see some connection attempts from
>places, including what is supposedly a network security company "
>protected.javapipe.com" but nobody getting in any more. So, I think the
>Chinese illegally accessed the system by cracking the root password and
>because I had naively left root access open on ssh, thinking an 18
>character, complex password would be sufficient.
>
>Thanks, back to $JOB.
>Tim
>
>
>On Mon, Jul 18, 2016 at 9:00 AM, Matt Flyer via TriLUG
><trilug at trilug.org>
>wrote:
>
>> Here is the list of commands that I would recommend running to try to
>> cross correlate the open connections via a process:
>>
>> (run each individually and save the output) ps acxfwwwe, lsof -Pwln,
>> and netstat -anpe
>>
>> You should also scour your log files (consider running them through
>> logwatch.
>>
>> Ultimately, I think you will want to rebuild the system image, but I
>> also think it is vitally important to try to identify how they got in
>> so that you can hopefully defend against it going forward.
>>
>> It is kind of hard to tell from the LSOF output, but it looks like
>they
>> may have launched copies of SSHD as root, which would mean a root
>level
>> compromise.
>>
>> Places like /tmp, which are relatively insecure are common locations
>> where you can find malware binaries.
>>
>> You could also try to run a chesksum (md5 or sha) of your system
>> binaries versus the repository to see if any of the system files,
>e.g.
>> ssh, have been replaced.
>>
>> Apache or other web servers are another common intrusion tactic,
>> especially if they can be made to do a remote download (remote file
>> inclusion I think it is called).
>>
>> On Mon, 2016-07-18 at 08:25 -0400, William Sutton via TriLUG wrote:
>> > anything in /var/log/secure?
>> >
>> > William Sutton
>> >
>> > On Mon, 18 Jul 2016, Tim Jowers via TriLUG wrote:
>> >
>> > >
>> > > Hi,
>> > >
>> > > I run these two less than a second apart:
>> > >
>> > > [root at test1 log]# fuser ssh/tcp
>> > >
>> > > ssh/tcp: 685 5066 5283 5284 5289 5290 5291
>529
>> > > 2 5293
>> > > 5294
>> > >
>> > > [root at test1 log]# fuser ssh/tcp
>> > >
>> > > ssh/tcp: 685 5066 5289 5290 5293 5294 5296
>529
>> > > 7 5298
>> > > 5299
>> > >
>> > >
>> > > Any ideas how to troubleshoot? I think I have some Chinese
>> > > search bot
>> > > malware based on this:
>> > >
>> > > [root at test1 log]# lsof -i
>> > >
>> > > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>> > >
>> > > sshd 685 root 3u IPv6 350221175 0t0 TCP *:ssh
>> > > (LISTEN)
>> > >
>> > > sshd 685 root 4u IPv4 350221177 0t0 TCP *:ssh
>> > > (LISTEN)
>> > >
>> > > mysqld 811 mysql 10u IPv4 350221673 0t0 TCP *:mysql
>> > > (LISTEN)
>> > >
>> > > sshd 5066 root 3r IPv4 4054471422 0t0 TCP
>> > > 198-20-184-56-host.colocrossing.com:ssh->
>> > > cpe-45-37-198-154.nc.res.rr.com:59006 (ESTABLISHED)
>> > >
>> > > sshd 5361 root 3r IPv4 4054875967 0t0 TCP
>> > > 198-20-184-57-host.colocrossing.com:ssh->112.85.42.99:41796
>> > > (ESTABLISHED)
>> > >
>> > > sshd 5362 sshd 3u IPv4 4054875967 0t0 TCP
>> > > 198-20-184-57-host.colocrossing.com:ssh->112.85.42.99:41796
>> > > (ESTABLISHED)
>> > >
>> > > sshd 5365 root 3r IPv4 4054877149 0t0 TCP
>> > > 198-20-184-56-host.colocrossing.com:ssh->112.85.42.99:openmailpxy
>> > > (ESTABLISHED)
>> > >
>> > > sshd 5366 sshd 3u IPv4 4054877149 0t0 TCP
>> > > 198-20-184-56-host.colocrossing.com:ssh->112.85.42.99:openmailpxy
>> > > (ESTABLISHED)
>> > >
>> > > sshd 5369 root 3r IPv4 4054886185 0t0 TCP
>> > > 198-20-184-56-host.colocrossing.com:ssh->221.229.172.99:36122
>> > > (ESTABLISHED)
>> > >
>> > > sshd 5370 sshd 3u IPv4 4054886185 0t0 TCP
>> > > 198-20-184-56-host.colocrossing.com:ssh->221.229.172.99:36122
>> > > (ESTABLISHED)
>> > >
>> > > sshd 5371 root 3r IPv4 4054886747 0t0 TCP
>> > > 198-20-184-57-host.colocrossing.com:ssh->221.229.172.99:15096
>> > > (ESTABLISHED)
>> > >
>> > > sshd 5372 sshd 3u IPv4 4054886747 0t0 TCP
>> > > 198-20-184-57-host.colocrossing.com:ssh->221.229.172.99:15096
>> > > (ESTABLISHED)
>> > >
>> > > java 18216 root 43u IPv6 3405192816 0t0 TCP
>> > > *:webcache
>> > > (LISTEN)
>> > >
>> > > java 18216 root 48u IPv6 3405192820 0t0 TCP *:8009
>> > > (LISTEN)
>> > >
>> > > java 18216 root 72u IPv6 3405192937 0t0 TCP
>> > > localhost.localdomain:8005 (LISTEN)
>> > >
>> > > httpd 26003 apache 3u IPv6 3253453758 0t0 TCP *:http
>> > > (LISTEN)
>> > >
>> > > httpd 26361 apache 3u IPv6 3253453758 0t0 TCP *:http
>> > > (LISTEN)
>> > >
>> > > httpd 27165 apache 3u IPv6 3253453758 0t0 TCP *:http
>> > > (LISTEN)
>> > >
>> > > httpd 27818 root 3u IPv6 3253453758 0t0 TCP *:http
>> > > (LISTEN)
>> > >
>> > > and
>> > >
>> > > [root at test1 log]# netstat -a
>> > >
>> > > Active Internet connections (servers and established)
>> > >
>> > > Proto Recv-Q Send-Q Local Address Foreign Address
>> > > State
>> > >
>> > > tcp 0 0 *:ssh *:*
>> > > LISTEN
>> > >
>> > > tcp 0 0 *:mysql *:*
>> > > LISTEN
>> > >
>> > > tcp 0 0 198-20-184-57-host.colo:ssh
>112.85.42.99:15265
>> > > ESTABLISHED
>> > >
>> > > tcp 0 0 198-20-184-56-host.colo:ssh
>> > > 221.229.172.99:48079
>> > > TIME_WAIT
>> > >
>> > > tcp 0 0 198-20-184-56-host.colo:ssh
>> > > 221.229.172.99:33195
>> > > ESTABLISHED
>> > >
>> > > tcp 0 0 198-20-184-57-host.colo:ssh
>> > > 221.229.172.99:44556
>> > > ESTABLISHED
>> > >
>> > > tcp 0 0 198-20-184-57-host.colo:ssh
>> > > 221.229.172.99:15096
>> > > TIME_WAIT
>> > >
>> > > tcp 0 608 198-20-184-56-host.colo:ssh cpe-45-37-198-
>> > > 154.nc.:59006
>> > > ESTABLISHED
>> > >
>> > > tcp 0 0 198-20-184-56-host.colo:ssh
>112.85.42.99:42180
>> > > ESTABLISHED
>> > >
>> > > tcp 0 0 *:webcache *:*
>> > > LISTEN
>> > >
>> > > tcp 0 0 *:http *:*
>> > > LISTEN
>> > >
>> > > tcp 0 0 *:ssh *:*
>> > > LISTEN
>> > >
>> > > tcp 0 0 localhost.localdomain:8005 *:*
>> > > LISTEN
>> > >
>> > > tcp 0 0 *:8009 *:*
>> > > LISTEN
>> > >
>> > > tcp 0 0 198-20-184-56-host.col:http
>ns336619.ip-37-187-
>> > > 16:18286
>> > > TIME_WAIT
>> > >
>> > > tcp 0 0 198-20-184-56-host.col:http
>> > > hydrogen081.a.ahrefs.:30831
>> > > TIME_WAIT
>> > >
>> > > and some StackOverflow article where someone posted that
>> > > *221.229.172.99*
>> > > is a Chinese search botnet.
>> > >
>> > > last and lastlog don't show anything. There is no
>/var/log/auth.log
>> > > present. Not sure if there should be. Just tried things based on
>> > > Internet
>> > > searching.
>> > >
>> > > I guess there is no easy way to kill this? Sounds like I should
>> > > just ask
>> > > for a new server instance (ChicagoVPS)? I use SVN to back up my
>> > > files there.
>> > >
>> > >
>> > > Thanks for any ideas.
>> > >
>> > > Tim
>> > > --
>> > > This message was sent to: William <william at trilug.org>
>> > > To unsubscribe, send a blank message to trilug-leave at trilug.org
>> > > from that address.
>> > > TriLUG mailing list :
>http://www.trilug.org/mailman/listinfo/trilug
>> > > Unsubscribe or edit options on the web : http://www.trilug.o
>> > > rg/mailman/options/trilug/william%40trilug.org
>> > > Welcome to TriLUG: http://trilug.org/welcome
>> --
>> This message was sent to: timjowers <timjowers at gmail.com>
>> To unsubscribe, send a blank message to trilug-leave at trilug.org from
>that
>> address.
>> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
>> Unsubscribe or edit options on the web :
>> http://www.trilug.org/mailman/options/trilug/timjowers%40gmail.com
>> Welcome to TriLUG: http://trilug.org/welcome
>>
>--
>This message was sent to: Roger <rogerb at bronord.com>
>To unsubscribe, send a blank message to trilug-leave at trilug.org from
>that address.
>TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
>Unsubscribe or edit options on the web :
>http://www.trilug.org/mailman/options/trilug/rogerb%40bronord.com
>Welcome to TriLUG: http://trilug.org/welcome
More information about the TriLUG
mailing list