Wireshark Traps For Young Players

I ran into an interesting problem in work today.  An Ethernet switch appeared not to be behaving correctly: frames that I know were being received on an 802.1q trunk (coming in from a WAN carrier) weren't being sent out an access port for no reason I could see.  The switch configuration was fine...there was no obvious reason it shouldn't have been working.

A bit of background.  

The IEEE 802.1q header has a bit in it call the CFI bit.  CFI stands for "Canonical Format Indicator". It is (or was) used to distinguish between Ethernet-format MAC addresses ("canonical") and Token Ring-format MAC addresses ("non-canonical").  Basically, this bit is set to 0 on Ethernet frames and 1 on Token Ring frames.  Since Token Ring is (mercifully) now a thing of the dim-and-distant past, this bit is now always set to 0 (which is why most people - including me - have never noticed it and may be oblivious to its existence).  The 802.1q standard specifies that frames received with the CFI bit set (indicating Token Ring frames) should not be transmitted on untagged Ethernet interfaces (because the MAC address format is different and won't make any sense).  This will become important later.

The more recent 802.1ad specification which - among other things - codifies "Q-in-Q" (a.k.a. double-tagging) renames and repurposes this bit so it is now called the DEI bit.  This is where my problems started, but more on that later.  DEI stands for "Discard Eligible Indicator" and is a hint that this packet can be discarded in the event of congestion on a link.  It is in the same bit position as the CFI bit was...it just has a completely different meaning now.  

This gets me to my first problem.  I was told that my carrier was sending me tagged frames with the DEI bit set (there was a legitimate reason for this).  I figured that my Ethernet switch might interpret the DEI bit as a CFI bit (remember, same bit position in the 802.1q header) and might (correctly, from its point of view) refuse to send the frame on an untagged Ethernet interface.  Sure enough, once the carrier stopped setting the DEI bit, everything started working OK.

That would be the end of the story except that when I ran packet-captures to see what was what was going on, the frames coming in from the carrier had the DEI bit clear.  I can think of a few possibilities:-

• My tip-off that the carrier was setting the DEI was incorrect and there was something else going on. Not likely since the problem went away as soon as the carrier changed this (and my theory around this seemed pretty sound).
• The interface-mirroring on the switch was "lying" about the CFI/DEI bit being set (which didn't seem hugely likely either)
• There was a problem with my packet-captures.

After some research, I found this (short but very informative) thread https://lists.linuxfoundation.org/pipermail/bridge/2016-March/009940.html.  The TL;DR version is that the Linux kernel "abuses" the CFI/DEI bit for its own internal purposes and always clears it !  This was fine when the bit was CFI (and therefore always 0), but it is distinctly not fine now that the bit should be interpreted as DEI.  As it happened, I had done all of my packet-captures on Linux machines 😞

So I set up an experiment.  I have a machine transmitting Ethernet frames (using this https://gist.github.com/austinmarton/1922600) with with the following characteristics:-

• IEEE 802.1q encapsulation (Ethertype 0x8100)
• VLAN tag 120
• CFI bit in the 802.1q header set to 1

Then I used Wireshark to capture these frames on a few platforms.  The results were eye-opening.

Linux

Sure enough, the 802.1q header and the VLAN tag are there, but the CFI/DEI bit is set to 0.  Bingo !!

Windows

The second packet-capture is done on the same NIC on the same laptop (it dual-boots Windows and Linux)

Even worse !!  Under Windows, I don't even get to the the 802.1q header.  I tried this on two different Windows machines (both Windows 10, admittedly) with the same results.  This could really send you off on a wild goose-chase if you were trying to troubleshoot a real problem.

Mac

Perfect: I can see the 802.1q header and the CFI/DEI bit is still set to 1, just as it should be.

The message thread indicates that this only happens with if the Ethertype value is 0x8100 (802.1q) or 0x88a8 (802.1ad) but not if another Ethertype is used.  Let's try that:-

Look at that !  Changing the Ethertype from 0x8100 to 0x8101 and now the Linux laptop leaves the CFI/DEI bit untouched (I had to twist Wireshark's arm to get it to parse 0x8101 as an 802.1q frame).  That makes sense: the Linux kernel no longer sees the frame as an 802.1q/802.1ad frame and therefore doesn't see the following two bytes as an 802.1q/802.1ad header that it can mess with.

The first lesson is that one needs to be very circumspect about packet-captures unless you are very sure about the behaviour of the platform you are doing the capturing on.

Second, if you have a WAN provider setting that DEI bit (e.g. on a "best-effort" WAN service), be aware that your switch may be refuse to pass those frames through.