[TriLUG] Web Usage Tracking...

Jon Carnes jonc at nc.rr.com
Fri Sep 28 20:08:47 EDT 2001


I originally set it up to do hourly dumps, since I had no idea how BIG each
file would get and I was worried that something might happen to suspend the
measurements.  The hourly start/stop was to help with both measures.

As an experiment today, I ran a second process that just collected the whole
work-day's traffic.  That file ended up a whopping ~200Mb.

The hourly script records all processes running "netstat -Mnc", then it
starts a new collection process, and then immediately stops the old process
(or processes).

The name of the files is given by:
HR_LOG=  `date --date '+1 hour' '+%b_%_d_%H'`.txt



----- Original Message -----
From: "Knowles, Christopher" <CKnowles at sumitomoelectric.com>
To: <trilug at trilug.org>
Sent: Friday, September 28, 2001 8:06 AM
Subject: Re: [TriLUG] Web Usage Tracking...


> netstat -Mnc  OK, now I understand.  two more questions, and then I'll
> wander off and mull things over.
>
> you say hourly log files... do you start netstat and then kill it before
> starting the next hour, or is there some special and more elegant way?
>
> Um, while thinking over my second question, I thought of an answer... so
> I'll leave it at one!
>
> Thanks again.
>
> CJK
>
> ----------  Forwarded Message  ----------
>
> Subject: Re: [TriLUG] Web Usage Tracking...
> Date: Fri, 28 Sep 2001 01:58:26 -0500
> From: "Jon Carnes" <jonc at nc.rr.com>
> To: <trilug at trilug.org>
>
> > On Thursday 27 September 2001 06:24 pm, you wrote:
> > > > You could set ipchains to log all outgoing packets destined for
ports
> > > > 80, 443, and 8008.  This would give you basic traffic analysis.  Not
> > > > perfect or elegant, but it's quick and easy.
> > >
> > > Heck, go ahead and log everything, but make sure you put in a scsi
disk
> > > sub-system for the firewall or your performance is going to take a
hit.
> > >
> > > I have a couple of firewalls based on RH6.2 and they work great for
>
> logging
>
> > > and I can even tell in real time what is going on:
> > >   netstat -Mn |grep 80
> > > tells me who is using the web and for what.  I have cron jobs pull the
>
> top
>
> > > 10's out of my logs nightly and then flush the logs after seven days.
> >
> > <SNIP>
> >
> > OK, can you give me an example of the cron job you use for that?  (I
>
> assume
>
> > the logs to which you refer are the packet logs...)
>
> Actually for logs, I pump netstat -Mnc into hourly log files that end with
> .txt.  To process them I use the following script:
> ===
> echo Top Ten http users
> cat *.txt |grep " 80 "|cut -c15-32 |sort |uniq -c |sort -g |tail -10
>
> echo Top Ten http destinations
> cat *.txt |grep " 80 "|cut -c36-55 |sort |uniq -c |sort -g |tail -10
>
> ===
> If you don't like IP's then you can pipe the IP's through:
> ... |nslookup |grep "Name"
>
> I also like to look at just the records from midnight to 6am.  This gives
me
> a good idea of the general Background traffic on the network.  This
traffic
> is mostly from folks who have left automated processes running on their
> boxes.  It is amazing how many connections we have that go to
weatherbug...
>
> Most folks just don't realize the cumulative consequences of their
actions.
> I had one lady here that had subscribed to over 16 automated services.
80%
> of my night-time traffic was being generated from her box.  Oddly enough,
> she also complained about how slow her box was compared to everyone
else...
> Well daaa. (sorry, its late.)
>
> Jon
>
> _______________________________________________
> TriLUG mailing list
> http://www.trilug.org/mailman/listinfo/trilug
>
> -------------------------------------------------------
> _______________________________________________
> TriLUG mailing list
> http://www.trilug.org/mailman/listinfo/trilug




More information about the TriLUG mailing list