1 / 27

Wise * TrafView Flow-based Measurement and Analysis System

This paper discusses the functionality of Wise*TrafView, a flow-based measurement and analysis system, and explores the need for more detailed analysis in today's networks. It also introduces enhanced application recognition mechanisms utilized by Wise*TrafView to accurately classify and analyze different types of internet applications.

gdebbie
Download Presentation

Wise * TrafView Flow-based Measurement and Analysis System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Wise*TrafViewFlow-based Measurement and Analysis System Pipefilters BOF, TIP2004 Jan. 27, 2004 Hyungseok Chung ETRI

  2. Topics • Backgrounds • Content-aware Application Recognition • Wise*TrafView Functionality • GUI Snapshots & Demonstration

  3. Common Measurement & Analysis Tools • RTT, Packet loss • Tool : ping, echoping, fping, gnuplotping, sting, … • System : Ping-ER, AMP, IWR - use ping internally • One-way delay, Packet delay • Designated measurement BOX • Surveyor, RIPE-NCC project, Smartbits • Route discovery • traceroute, skitter, mtr, ping plot, visualroute, neotrace,… • Misc : remote traceroute execution server • Throughput • netperf, iperf, treno, tcpblast, tcpspray, ttcp, pchar • Packet Capture & Analysis • tcpdump, Coralreef, cflowd, snoop, ethereal, … • Per Interface Traffic Volume and Errors • MRTG

  4. Why they are not enough? • Capturing Packets in Current Networks • High-speed networks (Mbps  Gbps  Tbps) • High-volume traffic • Streaming media (Windows Media, Real Media, Quicktime) • P2P traffic • Network Games • Network Security Attacks • Typical Flow-based Measurement • Non-flow based measurement is not enough for the above requirements • Typical Flow-based Measurement • Typically a flow is defined as a set of packets passing an observation point in the network during a certain time interval and having a set of common properties • 5-tuple packet header fields are used for this • New applications such as P2P, streaming and network games have characteristics of dynamic port allocation • More Detailed Analysis is needed • Typical Flow-based Measurement is not enough • Need more detailed analysis depending on applications • It may require content filtering

  5. Splitter Router Switch Network Operators Traffic Capture Agent raw streams of packets analysis result flow records Analysis Server How does Wise*TrafView work? AS 100 AS 200

  6. Application Recognition • Limitations of port-based recognition • The port database maintained by IANA doesn’t reflect real-world situation well • Most newer applications simply do not register their ports • Sometimes they even take advantage of well-known ports to pass thorough firewalls • Most bandwidth hogs, nowadays, dynamically allocate ports • They are not linked up with any fixed ports!

  7. Real-world Situation PosTech Traffic Breakdown • PosTech Campus Network • (24h sum in May, 304GB total volume)

  8. Enhanced Application Recognition • Wise*TrafView utilizes some enhanced proprietary recognition mechanisms in a comprehensive way • Internet Application Classification • signature matching • flow correlation • dynamic port recognition and utilization • some heuristics • Not only capable of discriminating applications, but also their sub-flows • e.g., HTTP  HTTP_REQ, HTTP_REP, HTTP_REQACK, etc.

  9. Internet Application Classification • Type S: Simple Application Type • for an application which uses a well-known port number or which uses a registered port number but are popularly used • Type P: Payload Application Type • for an application which uses a registered or ephemeral port number but requires payload inspections for precise classification • Type R: Reverse Application Type • for an application which uses a registered or ephemeral port number but requires comparison with a correlated reverse flow for the precise classification • Type C: Co-related Application Type • for an application which uses a dynamic port number assignment • Type U: Unknown Application Type • for applications which do not use registered port numbers and do not belong to any of the four types mentioned above

  10. Application Recognition Configuration Language (ARCL) application WWW { port_rep_name HTTP port 80 protocol TCP{ decision_group HTTP_REQ_REP_ACK { src_port >= 1024 dst_port == 80 } decision_group HTTP_REP_REQ_ACK { src_port == 80 dst_port >= 1024 }} port_rep_name HTTP_ALT port 8080 protocol TCP{ src_disc_pattern=="HTTP" in pkt 0-2 at byte 0 - 4 ( dst_disc_pattern=="GET" in pkt 0-3 at byte 0 - 10 || dst_disc_pattern=="POST" in pkt 0-3 at byte 0 - 10 ) decision_group HTTP_ALT_REQ_REP_ACK { src_port >= 1024 dst_port == 8080 } decision_group HTTP_ALT_REP_REQ_ACK { src_port == 8080 dst_port >= 1024 }} } application EDONKEY { port_rep_name EDONKEY_DOWN port 4662 protocol TCP{ dst_disc_pattern=="0xe33d000000" in pkt 2-3 at byte 0 - 4 decision_group EDONKEY_DOWN_REQ_REP_ACK { src_port >= 1024 dst_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 } decision_group EDONKEY_DOWN_REP_REQ_ACK { src_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 dst_port >= 1024 }} application FTP { port_rep_name FTP port 21 protocol TCP{ src_ref_pattern=="r/227 Entering Passive Mode \(\d{1,3},\d{1,3},\d{1,3},\d{1,3},(\d{1,4}),(\d{1,4})\)/$src_port = atoi($1)*1024 + atoi($2)" in pkt any at byte 0-35 induce FTP_DOWN_P decision_group FTP_REQ_REP_ACK { src_port >= 1024 dst_port == 21 } decision_group FTP_REP_REQ_ACK { src_port == 21 dst_port >= 1024 }} }

  11. Application Recognition Example % ftp server % ls % passive % quit % get wmggw.mp3 server.21 (FTP_CTRL_REQ) client.1302 server.21 (FTP_CTRL_REP) client.1302 49152 server.20 (FTP_DATA_DOWN) client.1303 client.1303 server.20 (FTP_DATA_UP) client.1306 server.49152 (FTP_DATA_PSV_UP) server.49152 (FTP_DATA_PSV_DOWN) client.1306 0 2 4 6 8 10 12 Time (sec)

  12. System Architecture Overview GUI Database ARCL Config-File Recognition and analysis Results (ODBC) Analysis Server Flow and packet Records (NFS) Capture Agent Capture Agent . . . IPCAP Card IPCAP Card NIC NIC . . . . . .

  13. Capturing Internet Traffic • Passive traffic capture • No side-effect imposed on any network devices and links • An optical or electric splitter, a.k.a. tap, is utilized • Wise*TrafView’s approach • Splitters + Packet Capture Card + High Performance Capture Engine • Adaptability maintained by supporting software-based capture as well • PCAP (Packet Capture) library • Doesn’t necessarily require a dedicated capture card; commonUnix boxes can substitute the cards • But yet, software-based capture is not equivalent to the card-based capture in terms of performance and functionality

  14. Specialized Packet Capture Devices • Link Signal Splitters • Electrical • Ethernet tap, DS-3 tap, etc. • Optical • ordinary optical splitter • independent of physical and data-link layer protocols • High Performance Packet Capture Cards • Model A: for lower speed links • Ethernet, FastEthernet, DS-3/(E3) • Model B: for middle speed links • ATM at OC-3, POS at OC-3, OC-12 (622Mbps), and GigaEthernet

  15. a flow a distinctive signature of application “X” Now, these pkts can also be identified as “X” Flow Concept • A “flow” is • a sequence of packets whose <src and dst IP addresses, src and dst port numbers, and protocol> are all identical • Why flow? • The size of entire raw packet streams for a given unit time are prohibitively enormous to be analyzed in time • Each individual packets in a flow contain duplicate information • Packets in the same flow are correlated; we can identify more packets which were previously categorized as unknown application a packet

  16. Agent Side: Generating Flow Records • Agents • carry on simple filtering and signature matching functions • generate flow records • This procedure aggregates and organizes the traffic information and reduces the amount of traffic volume transferred to the server

  17. Agent Structure

  18. Server Side:AS and Country Mapping • Identifying flow sources and destinations • Both source and destination IP address of a flow are mapped to ASes and finally to countries • This helps to locate the source and the sink of a flow • Discrimination among transit, inbound, and outbound traffic flows

  19. Analysis Server Structure

  20. Configurability and Adaptability • Why adaptability so important? • The highly frequenting nature of Internet applications’ appearance and disappearance • Swift mutation of applications • Localization of the use patterns of applications • Wise*TrafViewcopes with the problem by introducingARCL • By taking advantage of ARCL, Wise*TrafView • doesn’t need to be re-built or re-installed by any module for extension • can be easily reconfigured to handle a new application; modifying the configuration in ARCL and re-enforcing suffices

  21. The Major Functionality of Wise*TrafView • Transparent Packet Capture • complete independence of the existing networking equipment • Flow-based Measurement and Analysis • reduced load • higher degree of recognition • Understanding Application Specific Contexts • by means of enhanced application recognition algorithms • Scalable • can scale up from tens of Mbps to Gbps • supports various physical and data-link layer technologies • Highly Extensible and Adaptable • easy configuration with ARCL

  22. User Interface • Web-based Interface • simple • easy to use • intuitive • portable • A web site for each measurement site can be easily established • Authentication and authorization supported

  23. Visualization:Traffic Breakdown Report

  24. Visualization:Traffic Matrices

  25. Platforms • Hardware • For lower speed links (<= 622Mbps) • capture agent • high performance PC: 2 * P-III 1GHz+ CPU, 2GB+ RAM, 30GB+ HDD • analysis server • high performance PC: 2 * P-IV 1GHz+ CPU, 1GB+ RAM, 100GB+ HDD • For Higher speed links ( > 1 Gbps) • Dedicated Standalone Capture System • Hardwised logics for supporting wire-speed processing • Software • capture agent • Linux • analysis server • Linux, MySQL

  26. GPS Satellite ISP 1 Traffic Generator CDMA Basestation Traffic Center Traffic Generator ISP 2 Server ISP 3 Server Traffic Generator Server Measurement Agent Analysis Server Traffic Center IX Router A Possible Deployment Scenario

  27. Thank you ! Q & A Contact: Hyungseok Chung, Taesang Choi, Taesoo Jeong {chunghs, choits, tsjeong}@etri.re.kr

More Related