[TriLUG] Cautionary Tale: Intel/McAfee VSEL & Python Web Apps

[Rogers, Matthew via TriLUG]

If you want to talk IDS/Antimalware, checkout Cisco AMP.  Has Linux/Mac/Win support and runs off Talos.  Huge fan of Cisco lately in this area.  You manage it all via web console.  

http://www.cisco.com/c/en/us/products/security/fireamp-endpoints/index.html

Matthew Rogers
Senior Security Manager
O: +1.919.329.1130    M: +1.919.691.4636
allscripts.com | matthew.rogers2 at allscripts 

Allscripts: Building open, connected communities of health
_______________________________________________

-----Original Message-----
From: TriLUG [mailto:trilug-bounces+matthew.rogers2=allscripts.com at trilug.org] On Behalf Of Scott Chilcote via TriLUG
Sent: Friday, December 16, 2016 10:55 AM
To: Triangle Linux Users Group discussion list <trilug at trilug.org>
Subject: [TriLUG] Cautionary Tale: Intel/McAfee VSEL & Python Web Apps

Hi LUGgers,

VSEL is VirusScan Enterprise for Linux.  My employer provides some web applications based on a JavaScript/Python framework for our $CUSTOMER. 
$CUSTOMER mandates the use of Intel/McAfee VSEL for its RHEL webservers, which is not a bad idea in itself.  However, read on.

VSEL does active signature-based scanning of all scripts launched via Apache, among much else.  We discovered in early testing that the VSEL scanner processes had a noticeable impact on our script performance. 
This led to substantial troubleshooting.

We ran "top" and found that the VSEL scanner tasks were all at the upper end, pushing python way down.  Scanner tasks were getting around 60% of core utilization, python around 3%. 

Some of our scripts are CPU intensive for brief periods.  They read zip files in a repository and build hashes based on their contents.  So we spoke with $CUSTOMER's people who handle VSEL.  They decided to try adding "exclusions" for our web app's python scripts, which is how you tell VSEL to trust installed scripts. 

The performance immediately improved, so we moved on.

Some weeks further into the code cleanup and system tests, we find that VSEL is clobbering our scripts again.  It seems to have stopped honoring its exclusions.  The VSEL team runs a test using a "fake virus"
sentinel, and proves it.  They uninstall/reinstall VSEL, and we're good again.  Back to work.

A few weeks later with final tests approaching, VSEL interference is worse than ever.  Even low demand script operations, like reading directory contents are staggering. 

We go back to work with the VSEL people.  They uninstall/reinstall VSEL again.  Nope.  They add all of the data directories as exclusions ("just in case").  No help.  They add wildcard exclusions, removing swaths of directories and file types we barely touch.  No help.  They call Intel Security (who bought McAfee recently).  After they finally get through, Intel's support people start working down a "we think your dumb"
checklist of steps that we ruled out weeks ago. S-l-o-w-l-y.  Frustration!

We go through more weeks of trying stuff and head scratching.  Deadlines impend. 

Then we have that zen moment.  "Do we have *all* of the python scripts that the web applications use excluded?", they ask.  "Well yeah, all of the ones we wrote.  The entire web application directories are wild-carded."

"Are there others?"

"Well, sure.  The python API libraries.  Those are part of the system. 
Redhat installs those...!   Aren't they excluded by default??"

We aren't VSEL guys.  We aren't HBSS or active IDS people.  Why would VSEL waste time scanning the system libraries?  Sure, doing a signature check on them following a system update, that makes sense.  But would VSEL be designed to signature scan the system libs *every* *time* they were accessed by a running application?  Seriously?

As it turns out, McAfee thinks so. 

We added "/usr/lib64/python[2,3].[0-9]/.*/.*.py" to the exclusion list, which covers all of the system Python libraries.  The slowdown vanished.

We bought our own license for VSEL to help this problem (which costs beer money).  I'm not saying everyone should, working with Intel Security brings its own challenges.  But for us it was time to learn up.  My idea of a decent IDS was tripwire, until recently.  With the Linux webserver hacks that are widespread in the last 4-5 years, active intrusion detection on internet facing services will become much more common.  If it isn't already. 

I haven't needed to understand how it works until this huge time-waster fell into our laps.  I suspect we'll have even more learning to do as this project proceeds.

Scott C.

--
Scott Chilcote
scottchilcote at ncrrbiz.com
Cary, NC USA

--
This message was sent to: Matthew <matthew.rogers2 at allscripts.com> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
Unsubscribe or edit options on the web	: http://www.trilug.org/mailman/options/trilug/matthew.rogers2%40allscripts.com
Welcome to TriLUG: http://trilug.org/welcome