[TriLUG] help with fuser/ssh reporting lots of processes
Matt Flyer via TriLUG
trilug at trilug.org
Mon Jul 18 10:33:09 EDT 2016
As a follow up to this subject, I came across this post that I thought
discussed a unique concept: http://www.linuxquestions.org/questions/lin
ux-security-4/minihowto-using-openvpn-to-build-a-dwarvish-door-
4175582819/
In short, you expose (set to listen) your non public services, like SSH
only to a private address range. You then use OpenVPN, preferably
running on a nonstandard port, to grant access to that range. The
trick becomes to use the tls-auth option which makes OpenVPN go dark
unless the initial connection presents the proper certificate. This
creates a black hole where your services won't even show up on a port
scan: there is nothing to see here, keep moving.
On Mon, 2016-07-18 at 10:00 -0400, Matt Flyer via TriLUG wrote:
> Unless you have a reason to want connections from folks in that part
> of
> the world, I would consider adding the whole 221.224.0.0 -
> 221.231.255.255 range to your blacklist.
>
> Unfortunately, the IPV4 space is Swiss Cheese and it changes
> frequently, but finding some of the big blocks of the offenders space
> can help cut down on some of the noise.
>
> On Mon, 2016-07-18 at 09:47 -0400, Ken MacKenzie via TriLUG wrote:
> >
> > I get lots of those, I hope it goes without saying you should use
> > fail2ban
> > to catch repeat brute force offenders.
> >
> > I have 2 jail setups, a quick ban and a repeat offender ban. On
> > occasion I
> > check the logs and I add problem IP addresses to the blrules file
> > (I
> > use
> > shorewall for my firewall setup).
> >
> > /etc/shorewall/blrules is the black list file for shorewall. Once
> > in
> > there
> > it is a permanent ban on the offending IP address. I eventually
> > need
> > to
> > automate that process but for now the occasional continued trouble
> > maker is
> > not a big deal to handle manually.
> >
> > Ken
> >
> > On Mon, Jul 18, 2016 at 8:55 AM, Tim Jowers via TriLUG <trilug at tril
> > ug
> > .org>
> > wrote:
> >
> > >
> > >
> > > Thank you William and Roy,
> > >
> > > I see lots of things in /var/log/secure now! Tons of filed
> > > logins
> > > from
> > > China as I changed password and removed root from ssh login
> > > access.
> > > Duh! I
> > > know, slacker for not doing that before. Now my fuser ssh/tcp
> > > only
> > > shows
> > > one process and no others are showing up. Using an ssh key for
> > > ssh
> > > from now
> > > on and realize I can't be sloppy as crackers are out there. (I
> > > wonder if
> > > there is some lawsuit which could be filed against this outfit as
> > > I
> > > am sure
> > > they have US customers.)
> > > E.g. /var/log/secure has:
> > >
> > > Jul 18 10:02:12 test1 sshd[29252]: Failed password for root from
> > > 221.229.172.99 port 31660 ssh2
> > >
> > > Jul 18 10:02:14 test1 sshd[29254]: Failed password for root from
> > > 112.85.42.99 port 22571 ssh2
> > >
> > > Jul 18 10:02:14 test1 sshd[29252]: Failed password for root from
> > > 221.229.172.99 port 31660 ssh2
> > >
> > > Have a great week everyone,
> > >
> > > Tim
> > >
> > >
> > >
> > >
> > > On Mon, Jul 18, 2016 at 8:36 AM, Roy Vestal <rvestal at trilug.org>
> > > wrote:
> > >
> > > >
> > > >
> > > > Hi Tim,
> > > >
> > > > I did a quick whois on the 221.229.172.99 and found that is is
> > > > a
> > > > Chinese
> > > > IP:
> > > >
> > > > inetnum: 221.224.0.0 - 221.231.255.255
> > > > netname: CHINANET-JS
> > > > descr: CHINANET jiangsu province network
> > > > descr: China Telecom
> > > > descr: A12,Xin-Jie-Kou-Wai Street
> > > > descr: Beijing 100088
> > > > country: CN
> > > > admin-c: CH93-AP
> > > > tech-c: CJ186-AP
> > > > mnt-by: APNIC-HM
> > > > mnt-lower: MAINT-CHINANET-JS
> > > > mnt-routes: MAINT-CHINANET-JS
> > > > remarks: This object can only modify by APNIC hostmaster
> > > > remarks: If you wish to modify this object details
> > > > please
> > > > remarks: send email to hostmaster at apnic.net with your
> > > > remarks: organisation account name in the subject line.
> > > > status: ALLOCATED PORTABLE
> > > > source: APNIC
> > > > mnt-irt: IRT-CHINANET-CN
> > > > changed: hm-changed at apnic.net 20030626
> > > >
> > > > irt: IRT-CHINANET-CN
> > > > address: No.31 ,jingrong street,beijing
> > > > address: 100032
> > > > e-mail: anti-spam at ns.chinanet.cn.net
> > > > abuse-mailbox: anti-spam at ns.chinanet.cn.net
> > > > admin-c: CH93-AP
> > > > tech-c: CH93-AP
> > > > auth: # Filtered
> > > > mnt-by: MAINT-CHINANET
> > > > changed: anti-spam at ns.chinanet.cn.net 20101115
> > > > source: APNIC
> > > >
> > > > role: CHINANET JIANGSU
> > > > address: 260 Zhongyang Road,Nanjing 210037
> > > > country: CN
> > > > phone: +86-25-86588231
> > > > phone: +86-25-86588745
> > > > fax-no: +86-25-86588104
> > > > e-mail: ip at jsinfo.net
> > > > remarks: send anti-spam reports to spam at jsinfo.net
> > > > remarks: send abuse reports to abuse at jsinfo.net
> > > > remarks: times in GMT+8
> > > > admin-c: CH360-AP
> > > > tech-c: CS306-AP
> > > > tech-c: CN142-AP
> > > > nic-hdl: CJ186-AP
> > > > remarks: www.jsinfo.net
> > > > notify: ip at jsinfo.net
> > > > mnt-by: MAINT-CHINANET-JS
> > > > changed: dns at jsinfo.net 20090831
> > > > changed: ip at jsinfo.net 20090831
> > > > changed: hm-changed at apnic.net 20090901
> > > > source: APNIC
> > > > changed: hm-changed at apnic.net 20111114
> > > >
> > > > person: Chinanet Hostmaster
> > > > nic-hdl: CH93-AP
> > > > e-mail: anti-spam at ns.chinanet.cn.net
> > > > address: No.31 ,jingrong street,beijing
> > > > address: 100032
> > > > phone: +86-10-58501724
> > > > fax-no: +86-10-58501724
> > > > country: CN
> > > > changed: dingsy at cndata.com 20070416
> > > > changed: zhengzm at gsta.com 20140227
> > > > mnt-by: MAINT-CHINANET
> > > > source: APNIC
> > > >
> > > > % Information related to '221.228.0.0/14AS23650'
> > > >
> > > > route: 221.228.0.0/14
> > > > descr: CHINANET jiangsu province network
> > > > country: CN
> > > > origin: AS23650
> > > > mnt-by: MAINT-CHINANET-JS
> > > > changed: ip at jsinfo.net 20030630
> > > > source: APNIC
> > > >
> > > >
> > > > Try an lsof of each port and see what is using the tcp
> > > > connections:
> > > >
> > > > $> lsof -i :48079
> > > >
> > > > You should see the command, pid, and user that is using that
> > > > port. From
> > > > there you could use the lsof command again to see what spawned
> > > > the
> > > session:
> > > >
> > > >
> > > >
> > > > lsof -p PID (replace PID with the actual PID in the response)
> > > >
> > > >
> > > > From there you might be able to determine what is creating the
> > > > ssh
> > > > connection.
> > > >
> > > > HTH,
> > > >
> > > > -Roy
> > > >
> > > >
> > > >
> > > > On 7/18/16 8:19 AM, Tim Jowers via TriLUG wrote:
> > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I run these two less than a second apart:
> > > > >
> > > > > [root at test1 log]# fuser ssh/tcp
> > > > >
> > > > > ssh/tcp: 685 5066 5283 5284 5289 5290 529
> > > > > 1
> > > > > 5292
> > > 5293
> > > >
> > > >
> > > > >
> > > > >
> > > > > 5294
> > > > >
> > > > > [root at test1 log]# fuser ssh/tcp
> > > > >
> > > > > ssh/tcp: 685 5066 5289 5290 5293 5294 529
> > > > > 6
> > > > > 5297
> > > 5298
> > > >
> > > >
> > > > >
> > > > >
> > > > > 5299
> > > > >
> > > > >
> > > > > Any ideas how to troubleshoot? I think I have some
> > > > > Chinese
> > > > > search
> > > bot
> > > >
> > > >
> > > > >
> > > > >
> > > > > malware based on this:
> > > > >
> > > > > [root at test1 log]# lsof -i
> > > > >
> > > > > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
> > > > > NAME
> > > > >
> > > > > sshd 685 root 3u IPv6 350221175 0t0 TCP
> > > > > *:ssh
> > > > > (LISTEN)
> > > > >
> > > > > sshd 685 root 4u IPv4 350221177 0t0 TCP
> > > > > *:ssh
> > > > > (LISTEN)
> > > > >
> > > > > mysqld 811 mysql 10u IPv4 350221673 0t0 TCP
> > > > > *:mysql
> > > (LISTEN)
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > sshd 5066 root 3r IPv4 4054471422 0t0 TCP
> > > > > 198-20-184-56-host.colocrossing.com:ssh->
> > > > > cpe-45-37-198-154.nc.res.rr.com:59006 (ESTABLISHED)
> > > > >
> > > > > sshd 5361 root 3r IPv4 4054875967 0t0 TCP
> > > > > 198-20-184-57-host.colocrossing.com:ssh->112.85.42.99:41796
> > > (ESTABLISHED)
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > sshd 5362 sshd 3u IPv4 4054875967 0t0 TCP
> > > > > 198-20-184-57-host.colocrossing.com:ssh->112.85.42.99:41796
> > > (ESTABLISHED)
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > sshd 5365 root 3r IPv4 4054877149 0t0 TCP
> > > > > 198-20-184-56-host.colocrossing.com:ssh-
> > > > > >
> > > > > > 112.85.42.99:openmailpxy
> > > > > (ESTABLISHED)
> > > > >
> > > > > sshd 5366 sshd 3u IPv4 4054877149 0t0 TCP
> > > > > 198-20-184-56-host.colocrossing.com:ssh-
> > > > > >
> > > > > > 112.85.42.99:openmailpxy
> > > > > (ESTABLISHED)
> > > > >
> > > > > sshd 5369 root 3r IPv4 4054886185 0t0 TCP
> > > > > 198-20-184-56-host.colocrossing.com:ssh->221.229.172.99:36122
> > > > > (ESTABLISHED)
> > > > >
> > > > > sshd 5370 sshd 3u IPv4 4054886185 0t0 TCP
> > > > > 198-20-184-56-host.colocrossing.com:ssh->221.229.172.99:36122
> > > > > (ESTABLISHED)
> > > > >
> > > > > sshd 5371 root 3r IPv4 4054886747 0t0 TCP
> > > > > 198-20-184-57-host.colocrossing.com:ssh->221.229.172.99:15096
> > > > > (ESTABLISHED)
> > > > >
> > > > > sshd 5372 sshd 3u IPv4 4054886747 0t0 TCP
> > > > > 198-20-184-57-host.colocrossing.com:ssh->221.229.172.99:15096
> > > > > (ESTABLISHED)
> > > > >
> > > > > java 18216 root 43u IPv6 3405192816 0t0 TCP
> > > > > *:webcache
> > > > > (LISTEN)
> > > > >
> > > > > java 18216 root 48u IPv6 3405192820 0t0 TCP
> > > > > *:8009
> > > (LISTEN)
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > java 18216 root 72u IPv6 3405192937 0t0 TCP
> > > > > localhost.localdomain:8005 (LISTEN)
> > > > >
> > > > > httpd 26003 apache 3u IPv6 3253453758 0t0 TCP
> > > > > *:http
> > > (LISTEN)
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > httpd 26361 apache 3u IPv6 3253453758 0t0 TCP
> > > > > *:http
> > > (LISTEN)
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > httpd 27165 apache 3u IPv6 3253453758 0t0 TCP
> > > > > *:http
> > > (LISTEN)
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > httpd 27818 root 3u IPv6 3253453758 0t0 TCP
> > > > > *:http
> > > (LISTEN)
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > and
> > > > >
> > > > > [root at test1 log]# netstat -a
> > > > >
> > > > > Active Internet connections (servers and established)
> > > > >
> > > > > Proto Recv-Q Send-Q Local Address Foreign
> > > > > Address
> > > > > State
> > > > >
> > > > > tcp 0 0 *:ssh *:*
> > > > > LISTEN
> > > > >
> > > > > tcp 0 0 *:mysql *:*
> > > > > LISTEN
> > > > >
> > > > > tcp 0 0 198-20-184-57-host.colo:ssh
> > > > > 112.85.42.99:15265
> > > > > ESTABLISHED
> > > > >
> > > > > tcp 0 0 198-20-184-56-host.colo:ssh
> > > > > 221.229.172.99:48079
> > > > > TIME_WAIT
> > > > >
> > > > > tcp 0 0 198-20-184-56-host.colo:ssh
> > > > > 221.229.172.99:33195
> > > > > ESTABLISHED
> > > > >
> > > > > tcp 0 0 198-20-184-57-host.colo:ssh
> > > > > 221.229.172.99:44556
> > > > > ESTABLISHED
> > > > >
> > > > > tcp 0 0 198-20-184-57-host.colo:ssh
> > > > > 221.229.172.99:15096
> > > > > TIME_WAIT
> > > > >
> > > > > tcp 0 608 198-20-184-56-host.colo:ssh
> > > > > cpe-45-37-198-154.nc.:59006
> > > > > ESTABLISHED
> > > > >
> > > > > tcp 0 0 198-20-184-56-host.colo:ssh
> > > > > 112.85.42.99:42180
> > > > > ESTABLISHED
> > > > >
> > > > > tcp 0 0 *:webcache *:*
> > > > > LISTEN
> > > > >
> > > > > tcp 0 0 *:http *:*
> > > > > LISTEN
> > > > >
> > > > > tcp 0 0 *:ssh *:*
> > > > > LISTEN
> > > > >
> > > > > tcp 0 0 localhost.localdomain:8005 *:*
> > > > > LISTEN
> > > > >
> > > > > tcp 0 0 *:8009 *:*
> > > > > LISTEN
> > > > >
> > > > > tcp 0 0 198-20-184-56-host.col:http
> > > > > ns336619.ip-37-187-16:18286
> > > > > TIME_WAIT
> > > > >
> > > > > tcp 0 0 198-20-184-56-host.col:http
> > > > > hydrogen081.a.ahrefs.:30831
> > > > > TIME_WAIT
> > > > >
> > > > > and some StackOverflow article where someone posted that
> > > *221.229.172.99*
> > > >
> > > >
> > > > >
> > > > >
> > > > > is a Chinese search botnet.
> > > > >
> > > > > last and lastlog don't show anything. There is no
> > > > > /var/log/auth.log
> > > > > present. Not sure if there should be. Just tried things based
> > > > > on
> > > Internet
> > > >
> > > >
> > > > >
> > > > >
> > > > > searching.
> > > > >
> > > > > I guess there is no easy way to kill this? Sounds like I
> > > > > should just
> > > ask
> > > >
> > > >
> > > > >
> > > > >
> > > > > for a new server instance (ChicagoVPS)? I use SVN to back up
> > > > > my
> > > > > files
> > > > > there.
> > > > >
> > > > >
> > > > > Thanks for any ideas.
> > > > >
> > > > > Tim
> > > > >
> > > --
> > > This message was sent to: Ken M. <ken at mack-z.com>
> > > To unsubscribe, send a blank message to trilug-leave at trilug.org
> > > from that
> > > address.
> > > TriLUG mailing list : http://www.trilug.org/mailman/listinfo/tril
> > > ug
> > > Unsubscribe or edit options on the web :
> > > http://www.trilug.org/mailman/options/trilug/ken%40mack-z.com
> > > Welcome to TriLUG: http://trilug.org/welcome
> > >
More information about the TriLUG
mailing list