The Barking Seal

Nightmares restoring mailboxes with BackupExec and Exchange 2007

19Jun

Author: ben Categories: Infrastructure, Ramblings Comments: 0

Imagine a world in which you had diligently backed up data in your Active Directory and Exchange environment every day, carefully labeling your tapes and filing them away to an off-site storage facility in case you one day encountered a server failure. Continuing in our Imaginationland, pretend that a real scenario arose in which you had none of your domain controllers, Exchange systems, backup servers, or any of the environment in which the backups were created, but you still need to recover some of your mail data. Would you take comfort in the knowledge that your recovery tapes were a short drive away?

If you’re using Symantec’s Backup Exec product, I wouldn’t be so sure. Read on for more of my rant on Symantec… and partially Microsoft.

Dear Ned

19Jun

Author: ned Categories: IT Management, Infrastructure, Ramblings Comments: 0

Here at Applied Trust, we’re often asked tricky IT questions – sometimes, we have answers that might be interesting to a larger audience. The “Dear Ned” podcast is our chance to share these IT infrastructure questions and answers. Larry Nelson from w3w3.com will be interviewing us for regular episodes throughout 2009.

Our first two “Dear Ned” episodes are already on-line and accessible over at w3w3.com! The first gives an introduction to the series and a discussion of the Conficker worm. The second is a followup to an earlier blog post, and addresses the question “I saw your blog recommending setting data center thermostats to 75°. Do you really do that? And if so, how’s that working out?”.

Do you have a tricky IT question? Submit it here and it may be the next Dear Ned topic!

A special thanks to our friend Don Wrege for writing and recording our truly wonderful Dear Ned jingle!

Tags: "Dear Ned", popular, rants

A Tangible (negative) Impact of the Sun/MySQL Acquistion

17May

Author: ned Categories: Infrastructure, Ramblings Comments: 1

MySQL didn’t escape the Sun acquisition unscathed… hopefully Oracle doesn’t make the same dumb mistakes.

I took (what I thought would be) a few minutes this afternoon to upgrade a group of production MySQL servers at Applied Trust. I started by following the same process I have followed for at least four or five years: browse to mysql.com, click on “Download”, and follow the links to the latest RPMs for my Linux distributions.

The download went as expected, with the consistent MySQL branding lulling me in to a false sense of ease – this was something I’ve done dozens of times. I shouted down the hall that I’d be ready to start grilling dinner in a few minutes. Next, I scheduled downtime, did the necessary change documentation, and brought one of the slave MySQL servers down – I was ready to upgrade the database. I typed sudo rpm -Uvh MySQL-*-5.1.34-0.rhel5 and my pleasant ride through upgrade-land came to a screeching halt:

Worst ITIL Book Ever by Gerard Blokdijk

08Dec

Author: ned Categories: Ramblings Comments: 1

I just survived a book that I am pretty sure was written by a computer, and regardless is probably the worst book I’ve ever read. Whoever wrote the clever algorithm that generated random ITIL-related text in this rotten book probably deserves a neck-medal, but surely their name isn’t Blokdijk. It appears that this horrible book, “ITIL IT Service Management: 100 Most Asked Questions, by Gerard Blokdijk,” was written by a real human, but I prefer to assume this was a result of a poor application of techology…

Like it or not, virtualization is in it for the long haul

19Nov

Author: ben Categories: IT Management, Ramblings, Security Comments: 0

The Lone Sysadmin, Bob Plankers, comments on the lack of vendor commitment to virtualization, and I fully concur. I see it especially with smaller, proprietary niche vendors. They’ll threaten to pull support entirely if their software is hosted on a virtual system.

The core problem, in my view, is that they just don’t understand it. It’s not limited to virtualization, either. There’s push back on patching (on all platforms), joining systems to a Windows domain, changing account passwords or privilege levels, even locating multiple systems on different subnets!

My take is that many of these vendors do not come from an IT background. They are experts in some field, identified a problem that needed automating, and hired somebody to write the code for them, with no understand of security or architectural implications. They’re often very good at solving the problems in that niche, but when that system plays a role in a larger enterprise there’s no understanding of the big picture.

Tags: rants, vendors

Walk, then run – thoughts about event value

11Oct

Author: trent Categories: Ramblings, Security Comments: 0

In his post On the difficulties of event correlation, Ben talks about how hard event correlation is – and I couldn’t agree with him more. In addition, I am often surprised about how many organizations blindly run down the path to adding more event collection to their environment before they understand the ones they already have.

A great example is IDS deployment. Along with the rest of the infosec establishment, I generally agree that enterprise-wide IDS can be an important part of a comprehensive infosec program. However, I often see that such deployments don’t succeed in the long run. Everyone is excited about it initially, but after a few months their interest wanes and the platform falls into a state of disrepair. Why? Is it because IDS data isn’t useful? Not at all. Instead, I think these are the drivers:

Poor event correlation. Ben makes some suggestions as to why, but the bottom line is that it’s really hard to use data that isn’t correlated with the rest of the environment.
Failure to budget staff resources to maintain the platform — there is an additive staff cost of deploying IDS, even if 7×24 monitoring is outsourced.
IDS events are not on the top of the list of “event value.” By “event value,” I mean that eventually, folks realize that there are more important events that they’re not capturing. Events like “server down,” “disk full,” or “network linked failed.” If they’re not reporting/handling these higher value events already, adding a high quantity of lower value events results in them being perceived as noise.

This last item is really important and apparently not obvious. The bottom line is this: before deploying a platform like an IDS, first make sure that you’re already capturing and managing the interesting, high-value events that currently exist in the environment. OS and infrastructure device logs are already easily available, take time to capture them centrally and use them. One that’s been mastered, then it’s reasonable to take on the IDS event stream.

Tags: rants, Security

On the difficulties of event correlation

10Oct

Author: ben Categories: IT Management, Security Comments: 2

You wouldn’t know it by the number of vendors and products on the market, but event management and log correlation is really, really hard. Despite the excellent work by folks like Anton Chuvakin, enlisting any support at all for log centralization, monitoring, auditing, and intrusion detection can be like pulling teeth.

Indeed, who can blame leadership for hating event management?

It’s extremely time intensive to do well
It’s expensive, even if you go with an open source platform
Maintenance is a lot of work, requiring lots of hardware (particularly storage) and expertise
It’s woefully inaccurate, and stunningly misleading in some cases

My biggest frustration? The lack of a standard format. Sure, the logging experts will point out that acronym-filled standards like the CEF (Common Event Format) or the WTEF (WebTrends Enhanced Format) are out there, but nobody uses them. Thus, it’s left as an exercise to the leader to normalize logs in to a universal format.

Moving this in to the real world for a moment, let’s ponder the challenges that this brings to an enterprise of, say, 5,000 employees. This enterprise likely has a lot of Windows servers running Windows-y applications like Active Directory, Sharepoint, and Exchange. Said organize probably has a few Unix or Linux systems around, spewing out syslog data. Lots of network devices are around generating firewall rule matches and error data, and there’s probably several proprietary applications logging directly to a localized database.

A “real” event correlation system would need to capture, centralize, normalize, audit, correlate, and alert on ALL of this data. It will require lots of maintenance as upgrades occur and storage requirements go. And don’t depend too much on the vendor – they’re probably too busy forgetting to install patches to worry about “centralized what”?

But guess what? It doesn’t matter – you have to do it[PDF].

That isn’t to say that it’s hopeless. The point is to find the strategy that works best for your organization. Maybe you just capture the critical event logs from Windows systems. Or perhaps you have a world class IDS with custom rules that capture log in events. Whatever the case, don’t try to bite off more than you can chew, or it’s bound to fail in the end.

Tags: Event Correlation, logs, popular, rants

Categories

Archives

Recent Posts

Recent Comments