Jake Kappus.comCertified Geek, Dad, MMA fan, and ocassional Pro Wrestler

I ran in into this issue while installing a new 5508 Wireless Lan Controller at a customer about a month ago.

When installing controllers I almost always put the controller either on the switch management VLAN or it’s own VLAN for scalablilty, with the AP’s on their own VLAN.

In this case, when the AP’s are moved to any other subnet other than the one with the WLC, whether they have already joined and received a config or not, find the controller, do the initial join process, then fail with “Invalid AC Message Type 4” and “Failed to handle capwap control message from controller” errors.   The failed AP is listed in the AP list, but is unconfigured and reboots constantly.

After 3 weeks and 4 levels of TAC engineers, we finally figured it out.

It appears that the WLC is having a problem establishing the DTLS Secure tunnel between the AP and the WLC when the destination mac address does not begin with 00.  The APs had a mac address, starting with 00 so when in the MGMT VLAN that were working. But since the gateway had a non-00 mac address, as soon as you put them in a different vlan, the destination mac address was non-00, and the DTLS session was failing.

As a workaround, we’ve changed the gateway to be an HSRP Standby address. Although we aren’t really running HSRP for redundant cores, adding the standby address to the vlan interface has allowed for the gateway to now have a mac address beginning with 00:00:0c, which has effectively allowed all of the APs to work.

Edit:  This issue has been fixed in MR 6.0.196.0, though I was told by a TAC engineer today that he felt that release was rushed out too soon and isn’t recommending it if you are only looking to fix this issue.  I’m holding out for 7.0 for this and other bugs I have registered with Cisco for my customers.

I guess I could only fight the inevitable so long.  My job now has me doing more Cisco work than I ever have before.  I didn’t realize how much I actually remembered about EIGRP routing and Multicast until recently I had to forklift a brand new network infrastructure in for a customer.  Just over the past few months I’ve worked with route/switch, wireless and voice technologies, even VMWare all at one customer.  And it’s gone really well considering I’ve only been working with Exchange and OCS for the past year and a half.

Now I guess I’m going to be stuck in the Cisco world for a while again.  In March I’m scheduled to go to Cisco voice training for Cisco Unified Call Manager and Cisco Voice.  And I’m going to be taking the CCVP as well.

I am however still going to be at Microsoft TechEd 2010 in New Orleans in June.  I’ll probably end up concentrating on Hyper-V, System Center and Exchange 2010 while I am there.

Change is good.