Solving Real-World Problems with Wireshark

1. Solving Real-World Problems with Wireshark Top-Down vs. Bottom-Up Approach By Emil Prysak

2. Introduction • NOC (Network Operations Center) Engineer at Weight Watchers • Resolving various networking problems • Wireshark used frequently • Using top-down and bottom-up approaches depending on the given symptoms • Two examples: • IP Address Loss  Bottom-Up Approach • RightFax Email Delay  Top-Down Approach

3. Overview of OSI Model • Application Layer – file transfer, email, network management, software • Presentation Layer – data representation, encryption, protocol conversion • Session Layer – managing connections between applications • Transport Layer – error recovery, flow control • Network Layer – routing, switching, packet sequencing, IP addressing • Data Link Layer – frame synchronization, error control, physical (MAC) addressing • Physical Layer – cabling, bit stream, electric signal

5. Troubleshooting Approaches for Networking Problems Top-Down Approach Bottom-Up Approach • From Application to Physical • Software  Hardware • Works best when problem is isolated to one user or device • From Physical to Application • Hardware  Software • Works best when problem is affecting multiple users or devices

6. Wireshark • Open source packet analyzer • Able to sniff out various types of network traffic • Can use custom display and capture filters • Can also use Perl compatible regular expressions • Helpful when using both top-down and bottom-up approaches • Version used: 1.8.1

7. IP Address Loss • Some users are losing network connectivity • “ipconfig/release” – “ipconfig/renew” gives temporary fix, problem reoccurs when computer restarts • Obtaining different addresses each • Bottom-up approach will be used • Involves physical connectivity • Multiple users affected

8. IP Address Loss:List of Components Involved • Core router • wwjer-ro6509-1 (Cisco Catalyst 6500 series, IOS) • Core switches • Wwjer-sw6509-1 (Cisco Catalyst 6500 series, CatOS) • Wwjer-sw6509-2 (Cisco Catalyst 6500 series, CatOS) • DHCP server “LICNAAD01” 10.80.40.34 • Wireshark server “LICPSHARK01” 10.80.40.41 • Version 1.8.1 • Multiple end users • Jack C-345 • IPs: 192.168.0.136 and 10.80.110.50 • Jack C-336 • IPs: 192.168.0.137 and 10.80.110.126

9. IP Address Loss: First Steps • Needed to wait until issue was confirmed by more users • All affected users appeared to be on vlan 110 • Catalyst Switched Port Analyzer (SPAN) • Allows certain type of traffic to be picked up by designated sniffing server (LICPSHARK01) • set span 110 4/17 session 1

10. IP Address Loss: Wireshark Trace • Both interfaces used for capture, encompassing users on both switches • Filter needed for DHCP Packets: • Udp port 67 or udp port 68

11. IP Address Loss: Wireshark Trace

12. IP Address Loss: Wireshark Trace • D-Link Wireless Router

13. IP Address Loss: Finding The Wireless Router • Router was not always on, had to wait until another user was affected • Find router with switching commands • Show cam 00:15:E9:F3:EF:B0 • Router was found at jack E-649 • DHCP was enabled, and giving off addresses before DHCP server could do so • Router was removed and problem resolved

14. RightFax Email Delay • RightFax • client software by OpenText • receives faxed documents, converts to email, and sends to destination mailbox • RightFax emails delayed by hours • TallyFax1 reported the issue first • RightFax runs on separate server (NYCPFAX) • Top-Down approach will be used • Related to RightFax application • Reported by one user

15. RightFax Email Delay: Sample Mail • Fax received by server at 3:05: PM • Email received by recipient at 7:38 PM • Approximately 4.5 hours of delay

16. RightFax Email Delay: List of Components Involved • Field Users sending faxes • RightFax server (NYCPFAX) 10.40.40.46 • SMTP server (NYAPSMTP03) 10.75.10.6 • Office 365 Cloud

17. RightFax Email Delay: First Steps • Ideally, configuration of RightFax would be checked for problems. • However, RightFax is an unfamiliar program, and not much was discovered. • Saw that outgoing server was NYAPSMTP03 (Application Layer) • SMTP logging was enabled to find a root cause

18. RightFax Email Delay: SMTP Log Results • Initial thoughts: HRBenefits mailbox was full, so no emails were getting through. • Not valid: HRbenefits mailbox was using only 2.12 GB of 25 GB space allocated

19. RightFax Email Delay: Wireshark Trace • Jump to Network Layer • Set up Wireshark trace on NYAPSMTP03 • Capture Filter: tcp port 25 • Display Filter: smtp.rsp.parametermatches “Mailbox full“

20. RightFax Email Delay: Results • Error message “452 Mailbox Full” was coming from 10.75.10.6 (NYAPSMTP03) • Further research found that the C:\inetpub\mailroot\Drop directory was full of unsent emails • Destination email address was incomplete, and default was not valid. • Once quota was hit, emails were bounced back to NYCPFAX, clogging queue on fax server.

21. RightFax Email Delay: Results

22. RightFax Email Delay: Solution • When WW switched from MS Exchange to Office365, email aliasing was not considered. • With local Exchange server, “TALLYFAX1” could translate to TALLYFAX1@weightwatchers.com • By sending to the Office365 using SMTP Relay, the destination email address was considered invalid, and not pick up from C:\inetpub\mailroot\Drop directory • Notification address had to be changed to full email address for each user • Queue of emails in Drop directory on NYAPSMTP03 and NYCPFAX had to be cleared out • After change was made, all faxes and notifications were sent on time.

23. Conclusion • Wireshark is extremely useful for troubleshooting network issues from both types of approaches • Network Layer is in the middle • Makes it equally accessible from both ends • Given more experience, a Divide and Conquer approach may be more beneficial • Select any layer desired • If layer has no issues, check layer above it. • If issues were found, check layer below it. • The lowest layer with errors is the culprit.