TCP header, Lego (tm) style

The older I get, the more lessons I seem to learn (or, not learn) over and over.  Have you ever seen TCP offload work correctly?  Of course not!  I’ve been bitten by a TCP offload (aka TCP Offload Engine or TOE) problem in just about every environment I’ve touched in the last 20 years, and sadly this week was no exception.

To make a long story short, we have a production vmware ESXi 4.1 host with both Linux (CentOS) and Windows Server 2008 guests.  No problems were reported (or measured) with the Linux guests, but the Win 2008 guests suffered from extremely choppy network connections, for common services like Remote Desktop and backups (including lost connections).  As you probably know, I’m big into actually investigating the underlying cause of a problem rather than randomly throwing darts at it, and as such I grabbed some packet traces with wireshark.  Check this out:

wireshark analysis of poor TCP connection

Ouch! That is super ugly (this is across a LAN, btw)!  How can you screw up a single TCP connection so badly in 6 feet of cable?  Probably not the cable (or network), sherlock.  It appears this is a “known problem.”    While this problem (described in the vmware article as “Network performance is very slow and connections drop intermittently”) seems contrained in the article to vmware guests running on a Windows host, I can attest to this occuring on both ESXi 4.0 Update 1 and ESXi 4.1 hosts with Windows guests.  After following the instructions in this vmware article to remedy the situation by disabling TCP offload on the Win 2008 guests, they exhibit downright snappy network performance.  Check out the improved trace results:

wireshark analysis after disabling TCP offload

Moral of the story:  TCP offload always sucks.  Turn it off on Windows Server 2008 vmware guests.

Goodie #1: Learn why version control is important for all businesses across the board.

Goodie #2: Get some assistance in deciding “Git or Subversion? Git or Subversion? Git…?”

Goodie #3 (otherwise known as the cherry on top): Meet Jim Turpin, one of our fabulous network engineers, who embodies the concept of multi-discipline to a T both inside and outside of the office.

Click here to read Q3 2010, and, as always, enjoy the treat!

We’d love to hear from you, so please post your comments and questions here.

Twitter is well known for great application-layer monitoring and instrumentation, so this gap in monitoring is a surprise. It exposes a common misconception among social software companies – that their server and network infrastructure is “covered” by their hosting provider.  As web applications scale to even 1/1000 the size of Twitter, software becomes critically interdependent on the underlying network. Infrastructure should be instrumented and monitored at least as closely as the software that depends on it.

For more The Barking Seal articles on monitoring and troubleshooting, see:

I should emphasize that this was our experience, and your mileage may vary. AppliedTrust is not a web design firm – we pride ourselves on infrastructure (servers, networks, security, performance, etc.). Still, I am tremendously impressed with Drupal and you are probably making a mistake if you are building a complex web site and haven’t considered it.

Looking back, since 2001 we have transitioned from static HTML (managed with GoLive), to Joomla, to WordPress (which we continue to use for this blog), to Drupal. Each transition has been a marked improvement, and today, I can’t imagine using anything except Drupal (for complex sites) or WordPress (for simple ones). In closing, here is a visual history of AppliedTrust’s web platform “evolution”:

As our regular readers know, AppliedTrust is looking for a great infrastructure engineer who wants to work in Boulder, Colorado. This role is a “Jack of all trades” within the broad field of Information Technology – they get to play with networks, servers, software, and security.  One ideal candidate for this job would be a graduating Computer Science or Engineering major who has experience with Windows and Linux system administration and doesn’t want to spend all day programming.  We would definitely also consider someone with more work experience.  If you are interested, or know of a good candidate, please check out our jobs page: http://www.appliedtrust.com/jobs

As discussed in detail by the Apache infrastructure team, a cross-site scripting vulnerability in Atlassian’s JIRA led to a full root account compromise on the ASF’s issue and request tracking server. If you don’t care to read the full story from the infrastructure team, the following sequence of events led to the compromise:

1. Attackers opened a new JIRA issue with a malicious tinyurl.com link that led to the JIRA page with an XSS vulnerability
2. Simultaneously, attackers launched a brute force attack on the JIRA login form
3. Several administrators clicked the tinyurl link, which compromised their cookies (giving the attackers JIRA admin access)
4. Attackers uploaded malicious a JAR file that collected JIRA passwords at login. One of the compromised passwords had also been used for a local account with full sudo privileges.

There’s more to the story, but those points capture the bulk of the attack.

This compromise interests me because it’s an explicit, targeted, successful attack against a security conscious and capable next-generation web technology team. Several techniques were used in this attack:

• Social engineering. The attackers opened an issue as if they were a trusted source posting a legitimate link. The Apache administrators trusted them.
• Web application security flaw. XSS is #2 on the OWASP top 10 list.
• Lack of vigilance. As the infrastructure team points out, the same password was used in a number of cases, and the JIRA user was overly privileged.

I hear a lot of grumbling when I highlight XSS vulnerabilities in a penetration testing report. “Is this really a serious problem?” and “we’re not a target” and “it doesn’t matter if they steal the cookie” are common complaints. Let’s face it – if the Apache team can be powned, we should all be wary.

Boulder is one of many communities vying to be the lucky recipient of this experimental network. As a city known for its smart people, progressive policies, and high tech companies, we are a great fit for a project like this. To capitalize on this, the City has stepped up efforts to get as many people as possible to vote, both by setting up a fan page on Facebook and by declaring this weekend “Boulder Fiber Weekend.”  Although nominations are being accepted until March 26, the City is hoping to have everyone vote before midnight on March 21.

This is an awesome opportunity for us to bring in more jobs, boost our local economy, and enhance communications across all sectors of our community. And, of course, having lightning-fast Internet speeds would be pretty sweet, too. So, what are you waiting for? Vote now!!



The simplest way to understand this vulnerability is with an example. Assume there is a stock trading website, S-trade, that anyone who signs up for an account can access. This site has functionality available for every account – including things like logging in, logging out, transferring money, purchasing stock, etc. Our hero in the scenario is Bob. Bob trusts S-trade to make his trades and keeps a portion of his portfolio there. Malice is our villain. Malice is not interested in trading stocks or other portfolio tasks, only wreaking havoc. Bob and Malice both have accounts on S-trade with basic functionality. S-trade uses all of the standard security measures meant to authenticate and protect users. There is session management in place, data sent to and from the site is encrypted, and strong password policies are enforced. These do not bother Malice one bit. All Malice must do is get Bob to click on a specially crafted link while Bob is logged in to F-trade (i.e. Bob’s cookies and session IDs have not expired). The specially crafted link can take advantage of any functionality that already exists in the application, but to keep things simple we’ll use the logout functionality as an example. When logged in, both Bob and Malice’s sessions use the same logout code. If you right-click on the link to logout, you might get something like this for URL location:

https://www.s-trade.com/session.php?action=logout

This section of code will undoubtedly check to see if the user is logged in or if the session has timed out. Once it determines if the session is valid, it will do whatever the rest of the code accomplishes. If Malice could get Bob to click the link above, it would log Bob out of his session, just like if Bob had clicked “Logout” himself. There are many ways for Malice to mask this link to Bob.

Malice can embed it in her own HTML page on her domain with an iframe that runs when the HTML is loaded:

<iframe src=” https://www.s-trade.com/session.php?action=logout “>

As long as Bob is logged in, this code will run.

Malice could also use traditional email phishing techniques to hook Bob on the line.

Now, logging Bob out might only be a minor inconvenience, but you can see the power behind this vulnerability. If there were similar functionality that made a stock purchase or withdrew money, Bob’s account could really be put in jeopardy. If the site has other OWASP vulnerabilities in place in addition to this, Bob is really screwed. CSRF hooks right in to a lot of the most common and dangerous attacks.

The problem here is that no other checks are done to prove that the user requesting this action is Bob. All it checks for is if Bob recently logged in on this machine. Web sites need to start going to further lengths to prove requests are generated by the authenticated user. There are five major steps needed to prevent CSRF attacks:

1. Require authentication in GET and POST parameters, not just cookies.
2. Check the HTTP “Referer” header and make sure it comes from S-trade (the Referer header can always be forged, but this small step will do some amount of good).
3. Further limit the lifetime of authentication cookies.
4. Require queries which cause transactions to include a one-time token.
5. Eliminate all XSS vulnerabilities.

With large, existing applications, CSRF can be hard to mitigate completely, but organizations that are planning to build new web applications should wire protection against this right into the code from the get go. This sort of attack is only going to get more and more common and proactive prevention is crucial.

I’ve picked five of the Principles behind the Agile Manifesto that are particularly applicable to our field – read on to see how they look from an IT infrastructure perspective:

1) Our highest priority is to satisfy the customer through early and continuous delivery of valuable infrastructure.
2) Welcome changing requirements, even late in deployment. Agile processes harness change for the customer’s competitive advantage.
3) Business people and developers must work together daily throughout the project.
4) Simplicity–the art of maximizing the amount of work not done–is essential.
5) Continuous attention to technical excellence and good design enhances agility.

1) “Our highest priority is to satisfy the customer through early and continuous delivery of valuable infrastructure. “
The technical and business requirements documented at the beginning of a project are never perfect, and sometimes not even close to what the end user really needs. Agile emphasizes a focus on gathering customer feedback early and often. Pilot new technologies before making a commitment. I have been part of many IT projects that have met every defined requirement and deadline, but completely failed in the end because users hated the technology. Deploy new technology in phases – find a small group of smart, friendly users who are willing to help test your project as you get new pieces of functionality working. Don’t get married to a hardware/software vendor too early. Wireless, two-factor, remote access, and mobile solutions are all hot candidates for an iterative, phased deployment where frequent user feedback is essential for the project to succeed.

2) “Welcome changing requirements, even late in deployment. Agile processes harness change for the customer’s competitive advantage.”
Agile software development is all about responding to change gracefully, and many system and network administrators could learn a lesson from this attitude. Activities like gathering requirements, documenting use cases, and project planning are essential, but they should never get in the way of “doing what’s right” for the end user. Processes should enable agility, not be used as a roadblock by administrators who don’t to deal with something new. The Visible Ops Handbook has a great quote on this topic: “Like brakes in a car, IT controls let you go faster!”

3) “Business people and engineers must work together daily throughout the project.”
As computer people, we get stereotyped as geeks who are bad at communicating. Let’s prove them wrong! Agile encourages the use of crossfunctional teams – this is our opportunity to help the business people and developers “get it right” before we end up supporting a mess of an application in production. Many IT departments suffer from “silo’d operations”, where the network folks don’t talk to the Windows folks, who in turn don’t talk to the Unix folks or the developers. As infrastructure engineers, we build and run the networks and servers – the “glue” that allows our organizations to function. We are in the position to help the diverse teams surrounding us work together.

4) “Simplicity–the art of maximizing the amount of work not done–is essential.”
Engineered complexity is the drug of choice for many network and system architects. It’s fun to play with bleeding-edge, challenging technologies, but those aren’t always the best use of time and money. It’s cool to build a system that can support millions of users, but not even worth thiking scalability about for a typical in-house app. Like the Mr. Miyagi said, “Go, find balance.” — balance complexity with manageability, monitoritability, and testability. Although this principle was written by people focused on development/implementation, it applies equally well to operations. Responsible change planning, testing, and documentation reduces unplanned work.

5) “Continuous attention to technical excellence and good design enhances agility.”
Agile focuses on customer interaction and working solutions over “comprehensive documentation”, but that doesn’t eliminate the need for professional behavior. Solutions that are hacked-together without planning and rigorous testing will fail miserably. I can’t tell you how many times I have seen a “temporary fix” ignored for months until it blew up in some sysadmin’s face! Ward Cunningham describes this issue as “Technical Debt,” and if you are an IT person who has not heard the term, you should read about it here. You accumulate financial debt by spending beyond your means – technical debt builds up when you deploy networks, servers, and services without an appropriate investment in solid architecture, testing, monitoring, backups, and reasonable amount of documentation. If you must take on a technical debt, every effort should be made to “pay it off” as soon as possible!

Bonus: “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”
Agile encourages picking and choosing individual processes and tools to best fit the team. Is this a cheap hedge? Maybe, but it works for us! For example, we don’t actually “stand up” in our various status/operations meetings, and we sure don’t use notecards or peer programming. Still, the principles above are useful to help set a “tone” of customer-collaboration, iterative release, and general “agility”. Take the “Agile Principles” with a grain of salt, but don’t be afraid to steal an idea or two from those programmers!!

Image credit to hickoryhollow113 via Flickr (Creative Commons).

I know, you love your network card. You installed Linux, the NIC was autodetected at first boot, and everything “Just Worked.” Your server has been happily providing services over the network ever since.

But what do you really know about your network card? Is it the culprit of slower performance for your CPU-intensive application? Could you benefit from any of its advanced capabilities? Today’s network interface cards offer a number of hidden gems to the savvy administrator. In this article we’ll learn some of the most important tricks to understanding your NIC in Linux.

The ifconfig command, used to configure the interface, is the most basic command in any administrator’s network toolkit. We use it for day-to-day management of networking.

$ /sbin/ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:0D:56:F8:99:34
 inet addr:192.168.7.8  Bcast:192.168.7.255  Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
 RX packets:393090428 errors:0 dropped:0 overruns:0 frame:0
 TX packets:416862423 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:380927523 (363.2 Mb)  TX bytes:1252846220 (1194.8 Mb)
 Interrupt:16

While ifconfig tells us important details like the interface status, IP address, and error information, we’re left with a lot of questions. What driver is it using? What is the speed and duplex? What features does it support, and which are we taking advantage of?

We can learn more information from the handy ethtool command.

$ sudo /sbin/ethtool eth0
Settings for eth0:
 Supported ports: [ MII ]
 Supported link modes:   10baseT/Half 10baseT/Full
 100baseT/Half 100baseT/Full
 1000baseT/Half 1000baseT/Full
 Supports auto-negotiation: Yes
 Advertised link modes:  10baseT/Half 10baseT/Full
 100baseT/Half 100baseT/Full
 1000baseT/Half 1000baseT/Full
 Advertised auto-negotiation: Yes
 Speed: 100Mb/s
 Duplex: Full
 Port: Twisted Pair
 PHYAD: 1
 Transceiver: internal
 Auto-negotiation: on
 Supports Wake-on: g
 Wake-on: d
 Current message level: 0x000000ff (255)
 Link detected: yes

Now we can see that eth0 autonegotiated a 100/Full link. (See our article on the risks of autonegotiation here). ethtool can also modify settings on the network card. More on that later.

In addition, use ethtool -i to identify driver details:

$ sudo /sbin/ethtool -i eth0
driver: tg3
version: 2.2
firmware-version:
bus-info: 02:00.0

Be cautious of using the alternative mii-tool in place of ethtool for examining network card status. Most drivers include the standard mii hooks, but some do not and on those NICs you’ll see an “operation not supported” message. Furthermore, many systems include a dated mii-tool that will incorrectly report Gigabit speeds.

The lspci command examines all the PCI buses (conventional PCI, PCI-X, PCI Express) and reports information about connected devices, including the NICs. Run “sudo lspci -vv | less” and search for “Ethernet” for best results. The output on my example system looks like:

02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02)
 Subsystem: Dell Computer Corporation: Unknown device 014a
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
 Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
 Latency: 64 (16000ns min)
 Interrupt: pin A routed to IRQ 16
 Region 0: Memory at fcf30000 (64-bit, non-prefetchable) [size=64K]
 Region 2: Memory at fcf20000 (64-bit, non-prefetchable) [size=64K]
 Capabilities: [40] PCI-X non-bridge device.
 Command: DPERE- ERO- RBC=2 OST=0
 Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-      Capabilities
: [48] Power Management version 2
 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
 Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
 Capabilities: [50] Vital Product Data
 Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
 Address: 5494806804400880  Data: a222

lspci reveals a goldmine of Ethernet goodies. This system uses the very common Broadcom BCM5704 NIC, an enterprise-grade card from a company with questionable business ethics. It uses interrupt 16 (also available in the ifconfig output), lives on the PCI-X bus, and supports Message Signalled Interrupts, an alternative to traditional pin-based interrupts. This card also supports Vital Product Data, a collection of still more information that is available through the lsvpd command. You’ll often see unfamiliar information in the Capabilities section of the lspci output. Google is your friend.

dmesg provides still more detail about your NIC:

$ dmesg | grep eth0
divert: allocating divert_blk for eth0
eth0: Tigon3 [partno(BCM95704A6) rev 2002 PHY(5704)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:56:fe:69:34
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is off for TX and off for RX.

tg3 is a driver commonly used by Broadcom NICs. Here it reports a 100/full link, corroborating ethtool’s findings.

Now we know all about the NIC, but how is this information useful? A few ideas:

• Are we using the best driver for this NIC? The driver that gives the best performance, has the fewest bugs, and best supports its hardware capabilities? Now that we know the chipset (BCM 5704) we can Google for alternative drivers.
• If this is a very busy host, is the NIC generating excessive interrupts? Check the number of interrupts/second with mpstat, then look in /proc/interrupts for the eth0 line (which maps to IRQ 16, information we learned via ifconfig and lspci). Reasonable interrupt values vary between systems, but if you’re north of 10,000 intr/sec it may be time to investigate mitigation options such as interrupt coalescing.

$ grep eth0 /proc/interrupts
16:  851306724          0   IO-APIC-level  eth0

• You may need to learn more about your NIC before enabling bonding/teaming to aggregate multiple interfaces for redundancy or increasing throughput.

Image credit to Jeff Kubina via Flickr, used here under Creative Commons.