I have recently gone through some life changes which means I have less time to blog than I’d like. I wanted to publish a new post every week or two, however with the transition I have been busier than usual. The good news is that when I can regularly post again, I should have a lot of new content to share! Sorry for the extended absence – I hope to be back in the swing of things shortly.
Detecting Rogue DHCP Servers Using an ASA
DHCP is a protocol used nearly everywhere to configure network parameters of host devices automatically. A DHCP server located on a LAN segment (or elsewhere and reached via DHCP relay) will typically send parameters for the following:
Client IP Address
Default Gateway
Subnet Mask
DNS Servers
A rogue DHCP server will configure hosts with bad information for various reasons. It may simply be a rogue residential router connected to the network backwards. Or it could be a villain intentionally redirecting host traffic to a man-in-the-middle to gather information, disrupt critical business traffic, or other nefarious reasons. Rogue DHCP servers will cause problems that are frustrating at best, and a critical security hazard at worst.
The good news is that they are fairly easy to detect with a packet capture if you are connected to the LAN in question. However, sometimes I have a need to shut these down from a distance and that’s where the packet capture functionality of a Cisco ASA comes in handy.
First, login to the ASA and get to the Privileged EXEC (#) prompt. Run the following command:
Cisco-ASA# capture $CAPTURE-NAME interface $IF-NAME match udp any eq bootps any
Note:
$CAPTURE-NAME = whatever you want to call this packet capture.
$IF-NAME = the interface that you want to capture packets through.
This will match on any source address UDP port 67, which is the port that a DHCP server will use.
Let this run for a bit and try to generate some DHCP traffic (bounce a port if you can).
Use the command ‘show capture’ to see currently active captures and see if any data has been captured.
Cisco-ASA# show capture
capture TEST-CAPTURE type raw-data interface INSIDE [Capturing – 2058 bytes]
match udp any eq bootps any
Now that we know data has been captured, let’s take a closer look.
Cisco-ASA# show capture TEST-CAPTURE
7 packets captured
1: 3:45:22.542727 802.1Q vlan#10 P0 0.0.0.0.68 > 255.255.255.255.67: udp 300
2: 3:45:22.565934 802.1Q vlan#10 P0 192.168.1.1.67 > 255.255.255.255.68: udp 548
3: 3:45:22.728508 802.1Q vlan#10 P0 0.0.0.0.68 > 255.255.255.255.67: udp 300
4: 3:45:22.786443 802.1Q vlan#10 P6 172.16.31.1.67 > 255.255.255.255.68: udp 300
5: 3:45:22.786519 802.1Q vlan#10 P6 172.16.31.1.67 > 255.255.255.255.68: udp 300
6: 3:45:23.823504 802.1Q vlan#10 P0 0.0.0.0.68 > 255.255.255.255.67: udp 300
7: 3:45:23.888702 802.1Q vlan#10 P6 172.16.31.1.67 > 255.255.255.255.68: udp 300
7 packets shown
In this example, 172.16.31.1 is a valid DHCP server. 192.168.1.1 however is not! That addressing scheme sounds like an out-of-the-box consumer-grade Netgear router. We can get more information by appending the word ‘detailed’ to the previous command.
Cisco-ASA# show capture TEST-CAPTURE detail
7 packets captured
1: 3:45:22.542727 e0c7.677a.a13c ffff.ffff.ffff 0x8100 Length: 346
802.1Q vlan#10 P0 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] udp 300 [tos 0x18] (ttl 255, id 10635)
2: 3:45:22.565934 b039.56ab.32ea ffff.ffff.ffff 0x8100 Length: 594
802.1Q vlan#10 P0 192.168.1.1.67 > 255.255.255.255.68: [udp sum ok] udp 548 (ttl 64, id 0)
3: 3:45:22.728508 e0c7.677a.a13c ffff.ffff.ffff 0x8100 Length: 346
802.1Q vlan#10 P0 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] udp 300 [tos 0x18] (ttl 255, id 10636)
4: 3:45:22.786443 40a6.7713.afb1 ffff.ffff.ffff 0x8100 Length: 346
802.1Q vlan#10 P6 172.16.31.1.67 > 255.255.255.255.68: [udp sum ok] udp 300 [ttl 1] (id 47344)
5: 3:45:22.786519 40a6.7713.afb1 ffff.ffff.ffff 0x8100 Length: 346
802.1Q vlan#10 P6 172.16.31.1.67 > 255.255.255.255.68: [udp sum ok] udp 300 [ttl 1] (id 47345)
6: 3:45:23.823504 e0c7.677a.a13c ffff.ffff.ffff 0x8100 Length: 346
802.1Q vlan#10 P0 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] udp 300 [tos 0x18] (ttl 255, id 10637)
7: 3:45:23.888702 40a6.7713.afb1 ffff.ffff.ffff 0x8100 Length: 346
802.1Q vlan#10 P6 172.16.31.1.67 > 255.255.255.255.68: [udp sum ok] udp 300 [ttl 1] (id 47348)
7 packets shown
This shows us that the MAC address of the 192.168.1.1 host (aka villainous rogue DHCP server) is b039.56ab.32ea. We can enter the OIU (first 6 characters of a MAC address) into a tool like the Wireshark OUI Lookup Tools and discover that this is a Netgear device. Tracking it on the LAN is easy and out of the scope of this post.
Of course, none of this would be required if you have properly configured features like DHCP snooping or 802.1X port authentication. I hope this helps you save some time troubleshooting in the future.
When Should You Automate Something?
Automation has been hot in the networking world for quite some time now – however I still see many people asking for advice on how to get started. If you don’t know of a project or something you can do for fun, think some more.
Others ask at what point they should start automating tasks. Some like to break the answer down to a simple equation where if (time to automate task) < (time to complete task manually), you should clearly automate.
def should_i_automate(time_to_do_manually, time_to_automate):
if time_to_do_manually > time_to_automate:
return “You should automate this task!”
else:
return “You should do this manually!”
This leaves out the intangible benefits of skills learned while automating tasks that can be leveraged in the future. It leaves out the hard-to-quantify human errors that are avoided when humans are removed from repetitive tasks. I don’t know about you, but I find coding to be challenging and rewarding as well. These are just a few reasons why you may automate something even when the time payout isn’t immediate. What are your thoughts?
PPPoE Is Not Supported with Failover Enabled (ASA) – Seriously
Cisco states that PPPoE is only supported on the 5500x ASA models in single routed mode without failover. (see here). When a telco tech came and replaced a DSL modem with a transparent bridge several hundred miles from my office, they gave me little choice but to challenge Cisco on that statement. I configured the VPDN group and tied it in with the outside interface and everything was working like it was before. Or was it?
First lesson learned – the ASA will allow you to configure PPPoE on an interface when in a failover pair. There are no warning lights, no flashing alarms, it will just let you do it. And it might even work, like it did for me.
Or you might end up in a strange situation where the standby firewall has gone through the PPPoE stages and now has a session up. In my case, this was identified with the following command:
‘show vpdn session state pppoe’ (or something like that)
My active firewall was at the PADI_SENT stage and was stuck there for a long time. My standby firewall was at the SESSION_UP stage and had an active Internet connection over the DSL line. This is perplexing to say the least. I fixed it by shutting the switchport that connected to the WAN interface on my standby firewall. Then I stripped the VPDN configuration from the outside interface on the active firewall and re-applied it. After confirming that the state was SESSION_UP on the active firewall, I no shut the port on the standby firewall. The end result was that the DSL connection was up on the active firewall and the standby firewall was failover ready. I thought I had out-smarted Cisco, I should have known better.
Lesson two – an ASA will disable failover if it boots up and has PPPoE attached to an interface.
Of course these lessons always happen at inconvenient times and/or places. I learned this when I was upgrading the ASA code. I had done it dozens of times and expected this time to be no different. I copied the new code over to the firewall pair, verified the sha-512 signature, changed the boot variables, wrote memory, and reloaded the standby unit. I watched the clock tick for several minutes. After 10 minutes and the active unit still reporting that the standby unit had failed, I knew something had gone wrong.
I got in touch with someone on-site who confirmed that both firewalls had green active lights. Fantastic. I had them immediately power down the secondary unit until I could get eyes on the console output during a boot sequence.
The firewall boot looked normal – strange. I logged in and found out that failover was OFF! This can cause some serious issues that are very hard to detect and troubleshoot (especially from hundreds of miles away). One example is that any device which uses the firewall as a default gateway now has two devices responding to their ARP request. This is a race condition and any device that caches an ARP reply from the secondary/should-be-standby-but-is-active unit is in for a bad time. Good luck getting outside of your subnet, I hope they don’t cache that MAC for too long.
When I tried to enable failover, I got the message:
PPPOE Client cannot be enabled on interface, Gi1/1(outside)
failover is not compatible with above configurations,
user must manually remove or fix them as instructed before failover can be enabled.
The fix for this was to remove the ‘ip address’ and ‘pppoe client’ commands from the interface mentioned in the error message, then try again. I ended up removing those commands from both firewalls before finally enabling failover on the secondary unit, which did sync and settle into standby.
TL;DR – don’t configure PPPoE on an active/standby pair of ASAs.
Access Control Policies Won’t Apply to New Firepower Devices
I recently ran into a fairly simple issue during a Firepower installation. I registered some 5506-X Firepower modules to a Firepower Management Center (FMC) before I had any licensing applied to the FMC. I am using the classic licensing model, so this process may be different under smart licensing. After linking the control licenses with the FMC ID in the Cisco portal, I added the licenses to the FMC. The licenses I applied were being used, however the devices appeared in the FMC under ‘device management’ had no access control policy attached to them.
I remember setting access control policies when adding these devices, so I knew something was up. Click on the pencil icon on the right side of the device in question to edit that device. Click on the device tab to show general device information. If there is now adequate licensing for the device, you should be able to select different licensing schemes depending on what you have purchased and are using. The basic control license gives you ‘control’ and ‘protection’. Once those are both selected, your device will appear under the device management screen as having an access policy which can now be deployed.