Module 19

1. Module 19 WLAN Troubleshooting How To Make It Work Again

2. Objectives • Describe the WLAN troubleshooting process • Identify key concepts associated with troubleshooting WLAN Layer 1 and Layer 2 • Identify capacity related troubleshooting concepts • Identify and describe all diagnostic concepts and procedures • Describe performance troubleshooting • Describe how to use ADSP during a troubleshooting process

3. Troubleshooting Process

4. Overview

5. Define the Problem

6. Gather Information

7. Isolate the Fault

8. Troubleshooting Basics • Am I really in trouble? • Or is it by design? • What has/was changed recently? • Config • on controller • on target device • Logs • Check the datapath • VLANs • IP • Blocked by ACLs? (service pktcap on deny/drop) • Blocked by WIPS? (sh wireless wips client-blacklist) • Run packet traces

9. Troubleshooting

10. Connection Problem Checklist

11. Typical Physical Layer Issues • Space and Distance • Near/Far issues • Impenetrable objects • Mismatched power setting • Interference • Narrow Band • Wideband • Multipath 150 m

12. Typical Data Link Layer Issues • RF Created Issues • Hidden Node • Adjacent Cell • Co-channel interference • Roaming • Narrow Band • Wideband Collisions Occur within the circle

13. Adjacent Cell and Co-channel Interference • Occurs when the design does not take into account cells which can hear each other • Corrected by proper design and implementation

14. Roaming Issues • Occur when the design does not take into account coverage areas - Corrected by proper design and implementation Corrected by Proper Cell Design and Site-Survey Channel 1 Channel 11 Channel 11 Channel 6 Channel 1 Channel 1 Channel 11

15. Troubleshooting Capacity Issues

16. Capacity • Size of each cell • Minimum coverage • Maximum coverage • Number of clients served by each cell • At one time • At the same time • Total number of clients using system • Concurrently • Over-subscription • Types of traffic • Data • Voice

17. Troubleshooting AP Adoption

18. Layer 2 If Layer 2 Access Point adoption fails: • Verify the Access Point is connected to a local VLAN that has been extended to the Wireless Controller • Verify that no ACLs have been applied between the Wireless Controller and Access Point or directly to the Wireless Controller and Access Point that will drop EtherType 0x8783 • Verify the Ge port on the Access Point not been previously configured for 802.1Q tagging with a tagged native VLAN • Verify that Spanning Tree Protocol has not been enabled on the Ethernet Switch Port connected to the Access Point

19. Layer 3 If Layer 3 Access Point adoption fails: • Verify the Virtual Interface assigned to the Access Points Native VLAN has DHCP enabled • Verify the Access Points DHCP scope has DHCP options 189 or 191 defined • Verify the Wireless Controllers Native VLAN has a Virtual IP Interface defined with a valid IP Address • Verify the Wireless Controller has a default route assigned • Verify that no ACLs have been applied between the Wireless Controller and Access Point or directly to the Wireless Controller and Access Point that will drop UDP 24576 • Verify that a wired client on the Access Point VLAN can ping the management IP Address on the Wireless Controller • Verify that Spanning Tree Protocol has not been enabled on the Ethernet Switch Port connected to the Access Point

20. AP Adoption Policies • AP Adoption Policies can only assign user defined RF Domains and Profiles to new Access Points that have not been discovered by a Wireless Controller / Cluster • If an Access Point has been already adopted prior to creating an Adoption Policy, the Access Points will already be assigned to a default RF Domain and Profile • To assign a user defined RF Domain and Profile to an Access Point that has previously been discovered by the Wireless Controller / Cluster: • Disconnect the Access Points from the network • Remove the Access Points device configuration • Commit and Save the changes • When the Access Points are re-connected to the network, they will be automatically assigned to the user defined RF Domain and Profile during the adoption process • Do you have “Adopt If No Rules Match” enabled?

21. Profile and RF Domain Assignments group20-rfs4000# show wireless ap configured +---+----------------+-------------------+----------------+------------------+----------------+ |IDX| NAME | MAC | PROFILE | RF-DOMAIN | ADOPTED-BY | +---+----------------+-------------------+----------------+------------------+----------------+ | 1 | group20-ap650 | 00-23-68-31-14-2D | group20-ap650 | group20-rfdomain | un-adopted | | 2 | group20-ap7131 | 00-15-70-C7-8F-F0 | group20-ap7131 | group20-rfdomain | un-adopted | +---+----------------+-------------------+----------------+------------------+----------------+

22. Troubleshooting at the Device

23. Most Frequently Used Commands • Config • show running-config • <profile / policy / interface / etc> • device – full config for device • show cluster <config / status> • Events and Logging • show event-history • debug <category> <sub-category> • logging monitor <severity level> • Networking: • show ip interface brief • show interface switchport • show interface brief • show ipdhcp <…> • AP, Radio and WLAN-related • show adoption info • show adoption status • show wireless <ap configured / radio / wips / wlan / details> • show smart-rf history • Security Related • sh wireless wips client-blacklist • sh wireless wips history • service pktcap on deny/drop <ACL>

24. Debugging at the Device

25. WiNG5 Debugging Capabilities (only important ones)

26. Custom Event Filters

27. Core and Panic Snapshots Also an asterisk at the CLI prompt indicates presence of a Snapshot(s) in the flash:/snapshots directory: RFS6K-DC1*>

28. Logging at the Device

29. Enabling Logging at a Controller

30. Viewing Logs

31. Troubleshooting Datapath

32. Packet Captures • Each Wireless Controller and Access Point provides integrated packet capture facility • Captures wired and wireless traffic in real-time • Local & Remote captures • Wireless Packets can be captured encrypted or unencrypted • Captured flow may be • Streamed to the console • Saved to a file (local or remote using FTP) • Redirected to an external packet sniffer over IP using TZSP

33. Sniffer Redirect • Each radio on Dependent or Independent AP can be configured to redirect packets it sees in the air (on a specific channel) to a remote IP using the TaZman Sniffer Protocol (TZSP) • Such as Wireshark • All packets on the channel are forwarded as received by the radio • When enabled the Access Point radio cannot support Wireless Client traffic • Access Point must have an IP Address assigned TZSP Access Point Radio Host with IP Address 172.16.1.100

34. Trace Collection

35. Trace Collection Guidelines • A trace is only as good as the context in which it was captured. • Give the trace a descriptive name • Include date, MAC addr (if possible), Ticket#. • If wired trace, include location (svr / ap / etc.), trunk # • Should trace be L2 or L3? • Do not use capture filters; filtering can be performed later. • If possible, capture in continuous mode • Problems may take time to manifest

36. Trace Collection – Wireless Issue • Definition: Problem only occurs with MU / client. • Examples Include • Wireless Association failure • Roaming issue • Proxy ARP • Traces to collect • Wireless trace at client or AP • Collection from laptop at location of client • Collection at AP radio interface or wireless interface • Wired trace on segment client is on • Spanned switch port • Ge1 interface of AP using “pktcap” or “remote-debug” commands

37. Trace Collection – FW/Routing Issue • Definition: Involves two endpoints in separate IP domains. • Examples Include • Can’t access Internet (LAN →WAN) • VPN not working (LAN →WAN) • Outside can’t access internal server (WAN →LAN) • Traces to collect • Wired traces on each IP segment • Span port on each subnet to capture traffic from wireless AP (client traversal) and wired destination (server / voice gateway, etc.) • Wireless trace not required

38. Performance Troubleshooting

39. Checklist • Most of Performance problems are vague and transient: • You can’t do much of performance monitoring/troubleshooting without a full-scale management and monitoring system or a specialized solution.

40. Thresholds

41. Tools • Most of Performance problems are vague and transient: • You can’t do much of performance monitoring/troubleshooting without a full-scale management and monitoring system or a specialized solution. • Motorola Solutions recommends AirDefense Services Platform (ADSP) • Real-Time Troubleshooting and Proactive Prevention • Recording over 200 metrics for every device every minute • Spectrum Analysis and Interference Detection • Remote Packet Capture • Live View of RF Coverage and network/device performance • Client Connectivity Troubleshooting Module • Access Point Testing Module • Performance profiles, performance policy compliance monitoring and alarms • Historical Analysisand Troubleshooting • All recorded metrics are available for historical analysis with Advanced Forensics with drill-down • Rewind/Playback and Scoping • Alarms Forensics • Solve Issues Remotely • Level 1 Helpdesk Support • Mobile Laptop Analyzer

42. Things to looks for – 1 of 2 (ADSP)

43. Things to looks for – 2of 2 (ADSP)

44. Performance Profiles (ADSP)

45. Troubleshooting Using ADSP

46. LiveRF for Coverage Visualization LiveView for remote WLAN monitoring Real-Time and Historical Visualize application performance for: Voice Video Data Customized applications Visualize Application Performance

47. Define Performance Policy Define Alarms Severity Create custom performance policy compliance reports Automate performance report collection Configure ADSP to email the results to administrators every day Best Practice

48. Client Connectivity Designed for Level 1 Helpdesk Quickly Determine if it is a Wireless or Wired Network Issue Simple Debugging of Wireless Issues Escalation Only if Necessary Centralized Troubleshooting AP Connectivity Testing End-to-end Network Connectivity Testing from a Wireless Perspective Verify Access to Wireless Applications Servers Proactively Perform Network Tests Remote Troubleshooting Remote Location  • Data Center WAN ApplicationServer •  •   SecureServer •  DHCP Server

49. WLAN Troubleshooting – Review • Connection problems can occur due to inadequate coverage, RF interference, firmware mismatch, incorrect radio configuration, and client misconfiguration • Common Layer 1 issues in WLANs are: channel blockage due to Physical components, distance and RF interference, and differences in device power settings • Common Layer 2 issues in WLANs are: Hidden Nodes, Adjacent Cell interference, and interference created by a poor design. • Common roaming issues in WLANs are: poor design, not enough coverage between the cells, when there is too much coverage and clients bounce between cells, and when too many clients reside in the same location. These issues are corrected by proper cell design and site surveys.

50. WLAN Troubleshooting – Review • Identified capacity issues can be address by adding cells to address client connectivity requirements. • Debug logging can be enabled by using wireless controller debug mode • Core system events and panic snapshots process critical failures logged by the controller • Access points and controllers can utilize integrated packet capture facilities • Access points can redirect packets to a sniffer tool such as Wireshark over IP using the tazman sniffer protocol (TZSP) • ADSP has key troubleshooting modules including LiveRF and Advanced Troubleshooting. • LiveRF – visualizes wireless performance • Advanced Troubleshooting – remotely troubleshoots wireless connectivity issues