• Home
• About
• Contact
• Applied Trust

The Barking Seal

• Exploring your NIC for fun and profit

  12Jan
  Author: ben Categories: Infrastructure Comments: 0

  

  I know, you love your network card. You installed Linux, the NIC was autodetected at first boot, and everything “Just Worked.” Your server has been happily providing services over the network ever since.

  But what do you really know about your network card? Is it the culprit of slower performance for your CPU-intensive application? Could you benefit from any of its advanced capabilities? Today’s network interface cards offer a number of hidden gems to the savvy administrator. In this article we’ll learn some of the most important tricks to understanding your NIC in Linux.

  The ifconfig command, used to configure the interface, is the most basic command in any administrator’s network toolkit. We use it for day-to-day management of networking.

  $ /sbin/ifconfig eth0
  eth0      Link encap:Ethernet  HWaddr 00:0D:56:F8:99:34
   inet addr:192.168.7.8  Bcast:192.168.7.255  Mask:255.255.255.0
   UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
   RX packets:393090428 errors:0 dropped:0 overruns:0 frame:0
   TX packets:416862423 errors:0 dropped:0 overruns:0 carrier:0
   collisions:0 txqueuelen:1000
   RX bytes:380927523 (363.2 Mb)  TX bytes:1252846220 (1194.8 Mb)
   Interrupt:16

  While ifconfig tells us important details like the interface status, IP address, and error information, we’re left with a lot of questions. What driver is it using? What is the speed and duplex? What features does it support, and which are we taking advantage of?

  We can learn more information from the handy ethtool command.

  $ sudo /sbin/ethtool eth0
  Settings for eth0:
   Supported ports: [ MII ]
   Supported link modes:   10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Half 1000baseT/Full
   Supports auto-negotiation: Yes
   Advertised link modes:  10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Half 1000baseT/Full
   Advertised auto-negotiation: Yes
   Speed: 100Mb/s
   Duplex: Full
   Port: Twisted Pair
   PHYAD: 1
   Transceiver: internal
   Auto-negotiation: on
   Supports Wake-on: g
   Wake-on: d
   Current message level: 0x000000ff (255)
   Link detected: yes

  Now we can see that eth0 autonegotiated a 100/Full link. (See our article on the risks of autonegotiation here). ethtool can also modify settings on the network card. More on that later.

  In addition, use ethtool -i to identify driver details:

  $ sudo /sbin/ethtool -i eth0
  driver: tg3
  version: 2.2
  firmware-version:
  bus-info: 02:00.0

  Be cautious of using the alternative mii-tool in place of ethtool for examining network card status. Most drivers include the standard mii hooks, but some do not and on those NICs you’ll see an “operation not supported” message. Furthermore, many systems include a dated mii-tool that will incorrectly report Gigabit speeds.

  The lspci command examines all the PCI buses (conventional PCI, PCI-X, PCI Express) and reports information about connected devices, including the NICs. Run “sudo lspci -vv | less” and search for “Ethernet” for best results. The output on my example system looks like:

  02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02)
   Subsystem: Dell Computer Corporation: Unknown device 014a
   Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
   Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
   Latency: 64 (16000ns min)
   Interrupt: pin A routed to IRQ 16
   Region 0: Memory at fcf30000 (64-bit, non-prefetchable) [size=64K]
   Region 2: Memory at fcf20000 (64-bit, non-prefetchable) [size=64K]
   Capabilities: [40] PCI-X non-bridge device.
   Command: DPERE- ERO- RBC=2 OST=0
   Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-      Capabilities
  : [48] Power Management version 2
   Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
   Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
   Capabilities: [50] Vital Product Data
   Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
   Address: 5494806804400880  Data: a222

  lspci reveals a goldmine of Ethernet goodies. This system uses the very common Broadcom BCM5704 NIC, an enterprise-grade card from a company with questionable business ethics. It uses interrupt 16 (also available in the ifconfig output), lives on the PCI-X bus, and supports Message Signalled Interrupts, an alternative to traditional pin-based interrupts. This card also supports Vital Product Data, a collection of still more information that is available through the lsvpd command. You’ll often see unfamiliar information in the Capabilities section of the lspci output. Google is your friend.

  dmesg provides still more detail about your NIC:

  $ dmesg | grep eth0
  divert: allocating divert_blk for eth0
  eth0: Tigon3 [partno(BCM95704A6) rev 2002 PHY(5704)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:56:fe:69:34
  tg3: eth0: Link is up at 100 Mbps, full duplex.
  tg3: eth0: Flow control is off for TX and off for RX.

  tg3 is a driver commonly used by Broadcom NICs. Here it reports a 100/full link, corroborating ethtool’s findings.

  Now we know all about the NIC, but how is this information useful? A few ideas:

  • Are we using the best driver for this NIC? The driver that gives the best performance, has the fewest bugs, and best supports its hardware capabilities? Now that we know the chipset (BCM 5704) we can Google for alternative drivers.
  • If this is a very busy host, is the NIC generating excessive interrupts? Check the number of interrupts/second with mpstat, then look in /proc/interrupts for the eth0 line (which maps to IRQ 16, information we learned via ifconfig and lspci). Reasonable interrupt values vary between systems, but if you’re north of 10,000 intr/sec it may be time to investigate mitigation options such as interrupt coalescing.

  $ grep eth0 /proc/interrupts
  16:  851306724          0   IO-APIC-level  eth0

  • You may need to learn more about your NIC before enabling bonding/teaming to aggregate multiple interfaces for redundancy or increasing throughput.

  Image credit to Jeff Kubina via Flickr, used here under Creative Commons.

  
  Tags: linux, networking