Wireshark Traps For Young Players

I ran into an interesting problem in work today.  An Ethernet switch appeared not to be behaving correctly: frames that I know were being received on an 802.1q trunk (coming in from a WAN carrier) weren't being sent out an access port for no reason I could see.  The switch configuration was fine...there was no obvious reason it shouldn't have been working.

A bit of background.  

The IEEE 802.1q header has a bit in it call the CFI bit.  CFI stands for "Canonical Format Indicator". It is (or was) used to distinguish between Ethernet-format MAC addresses ("canonical") and Token Ring-format MAC addresses ("non-canonical").  Basically, this bit is set to 0 on Ethernet frames and 1 on Token Ring frames.  Since Token Ring is (mercifully) now a thing of the dim-and-distant past, this bit is now always set to 0 (which is why most people - including me - have never noticed it and may be oblivious to its existence).  The 802.1q standard specifies that frames received with the CFI bit set (indicating Token Ring frames) should not be transmitted on untagged Ethernet interfaces (because the MAC address format is different and won't make any sense).  This will become important later.

The more recent 802.1ad specification which - among other things - codifies "Q-in-Q" (a.k.a. double-tagging) renames and repurposes this bit so it is now called the DEI bit.  This is where my problems started, but more on that later.  DEI stands for "Discard Eligible Indicator" and is a hint that this packet can be discarded in the event of congestion on a link.  It is in the same bit position as the CFI bit was...it just has a completely different meaning now.  

This gets me to my first problem.  I was told that my carrier was sending me tagged frames with the DEI bit set (there was a legitimate reason for this).  I figured that my Ethernet switch might interpret the DEI bit as a CFI bit (remember, same bit position in the 802.1q header) and might (correctly, from its point of view) refuse to send the frame on an untagged Ethernet interface.  Sure enough, once the carrier stopped setting the DEI bit, everything started working OK.

That would be the end of the story except that when I ran packet-captures to see what was what was going on, the frames coming in from the carrier had the DEI bit clear.  I can think of a few possibilities:-

• My tip-off that the carrier was setting the DEI was incorrect and there was something else going on. Not likely since the problem went away as soon as the carrier changed this (and my theory around this seemed pretty sound).
• The interface-mirroring on the switch was "lying" about the CFI/DEI bit being set (which didn't seem hugely likely either)
• There was a problem with my packet-captures.

After some research, I found this (short but very informative) thread https://lists.linuxfoundation.org/pipermail/bridge/2016-March/009940.html.  The TL;DR version is that the Linux kernel "abuses" the CFI/DEI bit for its own internal purposes and always clears it !  This was fine when the bit was CFI (and therefore always 0), but it is distinctly not fine now that the bit should be interpreted as DEI.  As it happened, I had done all of my packet-captures on Linux machines 😞

So I set up an experiment.  I have a machine transmitting Ethernet frames (using this https://gist.github.com/austinmarton/1922600) with with the following characteristics:-

• IEEE 802.1q encapsulation (Ethertype 0x8100)
• VLAN tag 120
• CFI bit in the 802.1q header set to 1

Then I used Wireshark to capture these frames on a few platforms.  The results were eye-opening.

Linux

Sure enough, the 802.1q header and the VLAN tag are there, but the CFI/DEI bit is set to 0.  Bingo !!

Windows

The second packet-capture is done on the same NIC on the same laptop (it dual-boots Windows and Linux)

Even worse !!  Under Windows, I don't even get to the the 802.1q header.  I tried this on two different Windows machines (both Windows 10, admittedly) with the same results.  This could really send you off on a wild goose-chase if you were trying to troubleshoot a real problem.

Mac

Perfect: I can see the 802.1q header and the CFI/DEI bit is still set to 1, just as it should be.

The message thread indicates that this only happens with if the Ethertype value is 0x8100 (802.1q) or 0x88a8 (802.1ad) but not if another Ethertype is used.  Let's try that:-

Look at that !  Changing the Ethertype from 0x8100 to 0x8101 and now the Linux laptop leaves the CFI/DEI bit untouched (I had to twist Wireshark's arm to get it to parse 0x8101 as an 802.1q frame).  That makes sense: the Linux kernel no longer sees the frame as an 802.1q/802.1ad frame and therefore doesn't see the following two bytes as an 802.1q/802.1ad header that it can mess with.

The first lesson is that one needs to be very circumspect about packet-captures unless you are very sure about the behaviour of the platform you are doing the capturing on.

Second, if you have a WAN provider setting that DEI bit (e.g. on a "best-effort" WAN service), be aware that your switch may be refuse to pass those frames through.

The source of all network and security problems finally identified!!!

[This post is a slight departure from my stated policy of trying not to increase the average level of inane wittering on the Internet any further by keeping my opinions to myself in this blog, but this particular insight is just too penetrating not to share it with the world. Like all of my opinions, it is - of course - entirely correct ;-) ]


I have had an epiphany. I now know what causes pretty much all network problems: GUIs. At first blush, this may sound like a slightly sweeping statement but I have come to believe in it very deeply. Stick with me here and I will explain why.

Take today for example: Myself and one of my esteemed colleagues squandered an hour of our lives dealing with a guy in a secondary school where we support the Internet router. His Internet access was broken. I'll spare you the long and painful details of the hour...suffice it to say that by the end of it we determined that there was a server sitting between his 140 snotty, insolent teenagers and our router. It took a startlingly large fraction of that hour to glean from this barely-adequate specimen of humanity that this server even existed. We also figured out that - somehow - the server was at the core of the problem. After a conversation reminiscent of having teeth pulled, our hero volunteered that - infact - there had been a change to the server that morning: he had uninstalled Microsoft ISA server off it !! Somehow - and it completely eludes me how anyone can be quite this gormless - it never entered his head that perhaps uninstalling the proxy/firewall software off the server separating the hormonal masses from the Internet router might be somehow related to his current predicament (140 horny teenagers separated from their porn supply and becoming increasingly antsy about it).

So, how do I extrapolate from this to my theory about GUIs being the root of all evil ? Well, if that server had been a Linux server, there is no way on earth that this guy would have taken it upon himself to touch it. The slightly arcane (yes...I admit it) Linux command-line has a way of scaring off people like this who really need most of their brain power just so they remember to breathe regularly and are taking huge risks by trying to apply their limited stock of intelligence to anything else. In short, command-lines have a way of making things appear a little harder to do than they actually are and therefore act as a built-in safety-net, preventing "special" people from trying to do things they are simply not equipped to do. GUIs, have exactly the opposite effect: they allow the dimmest of knuckle-dragging troglodytes to poke and prod at things they don't really understand until eventually they manage to break it.

I formulated a more limited form of this theory some years ago when I formed the opinion that Checkpoint Firewall-1 was the source of all security problems on the Internet. When I first started working in the field of data security I could never really understand how hackers seemed to be able to waltz past the best of access lists and firewalls as if they weren't there. How could it be that the hackers were all so clever and the developers of firewalls were all apparently dribbling idiots ? Then, one day, I was on-site with a large multinational customer watching the guys in there trying to get an application working through a Checkpoint firewall. So, they fired up Checkpoint's very lovely GUI and they added the rule they thought should do the trick. It didn't, so the relaxed the rule a little further. It still didn't, so they relaxed it a little further again, and so the cycle continued through several iterations until eventually the application did work and the "firewall" was reduced to the functional equivalent of a piece of wire. At that moment I understood for the first time that security holes were rarely caused by weaknesses in firewalls and far more often caused by mental deficiencies in those charged with configuring them. I also understood that the GUI was at fault: Checkpoint's (lovely) GUI makes it very easy to set up rules without the bothersome inconvenience of having to have the remotest understanding of what the hell you are doing. If they had a PIX rather than a Checkpoint (these were the halcyon days before PIX Device Manager, when all was right with the world), this would not have happened. The only thing I didn't grasp at the time was exactly how generally-applicable the GUI theory was.

IPSec problems between Cisco PIX and WatchGuard Firebox

OK...this is my first useful post to this blog.

I have spent the better part of the day trying to diagnose an IPSec connectivity problem between a Cisco PIX (version 6.3.5) and a WatchGuard Firebox of some kind (I have no visibility of it). The problem turns out to be rather subtle, so I thought I would share it here.

The configuration on the PIX is fairly standard (IP addresses have been changed to protect the innocent !):-

crypto ipsec transform-set Esp3DesMD5 esp-3des esp-md5-hmac
:
:
access-list VPNToFirebox permit 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
:
:
crypto map VPN 60 ipsec-isakmp
crypto map VPN 60 match address VPNToFirebox
crypto map VPN 60 set pfs
crypto map VPN 60 set peer 2.2.2.2
crypto map VPN 60 set transform-set Esp3DesMD5
crypto map VPN 60 set security-association lifetime seconds 86400 kilobytes 8192
:
:
isakmp key ******** address 2.2.2.2 netmask 255.255.255.255 no-xauth no-config-mode


The ISAKMP (Phase 1) SA establishes just fine, but the IPSEC (Phase 2) SA never comes up. Watching the debug information on the PIX, here's what happens:-


ISAKMP : Checking IPSec proposal 1
ISAKMP: transform 1, ESP_3DES
ISAKMP: attributes in transform:
ISAKMP: SA life type in seconds
ISAKMP: SA life duration (VPI) of 0x0 0x1 0x51 0x80
ISAKMP: SA life type in kilobytes
ISAKMP: SA life duration (basic) of 8192
ISAKMP: group is 2
ISAKMP: encaps is 61433
ISAKMP: authenticator is HMAC-MD5
ISAKMP (0): atts not acceptable. Next payload is 0
ISAKMP (0): SA not acceptable!
ISAKMP (0): sending NOTIFY message 14 protocol 0
return status is IKMP_ERR_NO_RETRANS


The key is the "encaps is 61433" line. What this should say is "encaps is 61443" which is the (old, pre-RFC3947) encapsulation ID for ESP via NAT Traversal. As it is, the PIX has no idea what "61433" is supposed to be and the SA negotiation fails.

Here's what the debug output looks like when talking to another PIX (which sends the correct ID):-


ISAKMP (0): processing SA payload. message ID = 177759204
ISAKMP : Checking IPSec proposal 1
ISAKMP: transform 1, ESP_3DES
ISAKMP: attributes in transform:
ISAKMP: encaps is 61443
ISAKMP: SA life type in seconds
ISAKMP: SA life duration (VPI) of 0x0 0x1 0x51 0x80
ISAKMP: SA life type in kilobytes
ISAKMP: SA life duration (basic) of 8192
ISAKMP: authenticator is HMAC-MD5
ISAKMP: group is 1
ISAKMP (0): atts are acceptable.IPSEC(validate_proposal_request): proposal part #1,

Here the encapsulation ID is correct and the tunnel comes up.

The problem only arises when NAT is detected on the path between the Firebox and the PIX...otherwise there is no need to encapsulate the ESP inside UDP.

References

• Negotiation of NAT-Traversal in the IKE - http://tools.ietf.org/html/draft-ietf-ipsec-nat-t-ike-04