[TriLUG] Cautionary Tale: Intel/McAfee VSEL & Python Web Apps

[Scott Chilcote via TriLUG]

Hi LUGgers,

VSEL is VirusScan Enterprise for Linux.  My employer provides some web
applications based on a JavaScript/Python framework for our $CUSTOMER. 
$CUSTOMER mandates the use of Intel/McAfee VSEL for its RHEL webservers,
which is not a bad idea in itself.  However, read on.

VSEL does active signature-based scanning of all scripts launched via
Apache, among much else.  We discovered in early testing that the VSEL
scanner processes had a noticeable impact on our script performance. 
This led to substantial troubleshooting.

We ran "top" and found that the VSEL scanner tasks were all at the upper
end, pushing python way down.  Scanner tasks were getting around 60% of
core utilization, python around 3%. 

Some of our scripts are CPU intensive for brief periods.  They read zip
files in a repository and build hashes based on their contents.  So we
spoke with $CUSTOMER's people who handle VSEL.  They decided to try
adding "exclusions" for our web app's python scripts, which is how you
tell VSEL to trust installed scripts. 

The performance immediately improved, so we moved on.

Some weeks further into the code cleanup and system tests, we find that
VSEL is clobbering our scripts again.  It seems to have stopped honoring
its exclusions.  The VSEL team runs a test using a "fake virus"
sentinel, and proves it.  They uninstall/reinstall VSEL, and we're good
again.  Back to work.

A few weeks later with final tests approaching, VSEL interference is
worse than ever.  Even low demand script operations, like reading
directory contents are staggering. 

We go back to work with the VSEL people.  They uninstall/reinstall VSEL
again.  Nope.  They add all of the data directories as exclusions ("just
in case").  No help.  They add wildcard exclusions, removing swaths of
directories and file types we barely touch.  No help.  They call Intel
Security (who bought McAfee recently).  After they finally get through,
Intel's support people start working down a "we think your dumb"
checklist of steps that we ruled out weeks ago. S-l-o-w-l-y.  Frustration!

We go through more weeks of trying stuff and head scratching.  Deadlines
impend. 

Then we have that zen moment.  "Do we have *all* of the python scripts
that the web applications use excluded?", they ask.  "Well yeah, all of
the ones we wrote.  The entire web application directories are
wild-carded."

"Are there others?"

"Well, sure.  The python API libraries.  Those are part of the system. 
Redhat installs those...!   Aren't they excluded by default??"

We aren't VSEL guys.  We aren't HBSS or active IDS people.  Why would
VSEL waste time scanning the system libraries?  Sure, doing a signature
check on them following a system update, that makes sense.  But would
VSEL be designed to signature scan the system libs *every* *time* they
were accessed by a running application?  Seriously?

As it turns out, McAfee thinks so. 

We added "/usr/lib64/python[2,3].[0-9]/.*/.*.py" to the exclusion list,
which covers all of the system Python libraries.  The slowdown vanished.

We bought our own license for VSEL to help this problem (which costs
beer money).  I'm not saying everyone should, working with Intel
Security brings its own challenges.  But for us it was time to learn
up.  My idea of a decent IDS was tripwire, until recently.  With the
Linux webserver hacks that are widespread in the last 4-5 years, active
intrusion detection on internet facing services will become much more
common.  If it isn't already. 

I haven't needed to understand how it works until this huge time-waster
fell into our laps.  I suspect we'll have even more learning to do as
this project proceeds.

Scott C.

-- 
Scott Chilcote
scottchilcote at ncrrbiz.com
Cary, NC USA