The Barking Seal

On the difficulties of event correlation

10Oct
Author: ben Categories: IT Management, Security
You wouldn’t know it by the number of vendors and products on the market, but event management and log correlation is really, really hard. Despite the excellent work by folks like Anton Chuvakin, enlisting any support at all for log centralization, monitoring, auditing, and intrusion detection can be like pulling teeth.

Indeed, who can blame leadership for hating event management?
- It’s extremely time intensive to do well
- It’s expensive, even if you go with an open source platform
- Maintenance is a lot of work, requiring lots of hardware (particularly storage) and expertise
- It’s woefully inaccurate, and stunningly misleading in some cases
My biggest frustration? The lack of a standard format. Sure, the logging experts will point out that acronym-filled standards like the CEF (Common Event Format) or the WTEF (WebTrends Enhanced Format) are out there, but nobody uses them. Thus, it’s left as an exercise to the leader to normalize logs in to a universal format.

Moving this in to the real world for a moment, let’s ponder the challenges that this brings to an enterprise of, say, 5,000 employees. This enterprise likely has a lot of Windows servers running Windows-y applications like Active Directory, Sharepoint, and Exchange. Said organize probably has a few Unix or Linux systems around, spewing out syslog data. Lots of network devices are around generating firewall rule matches and error data, and there’s probably several proprietary applications logging directly to a localized database.

A “real” event correlation system would need to capture, centralize, normalize, audit, correlate, and alert on ALL of this data. It will require lots of maintenance as upgrades occur and storage requirements go. And don’t depend too much on the vendor - they’re probably too busy forgetting to install patches to worry about “centralized what”?

But guess what? It doesn’t matter - you have to do it[PDF].

That isn’t to say that it’s hopeless. The point is to find the strategy that works best for your organization. Maybe you just capture the critical event logs from Windows systems. Or perhaps you have a world class IDS with custom rules that capture log in events. Whatever the case, don’t try to bite off more than you can chew, or it’s bound to fail in the end.
Tags: Event Correlation, Logs, Popular, Rants