IPFIX/NetFlow Mediator Implementation and Test Results

1. IPFIX/NetFlow Mediator Implementationand Test Results 2007/3/22Daisuke Matsubara (Hitachi), Atsushi Kobayashi (NTT)

2. Overview • Background • IPFIX Mediator concept and draft was introduced in 65th and 67th IETF meeting. (draft-kobayashi-ipfix-mediator-01.txt) • Mediator allows us to monitor the overview traffic such as traffic matrix, and retrieve specific flow records anytime. • routers are able to simply export flows without aggregation even in large scale network, with minimum sampling rate. • Actual prototype implementation of IPFIX/NetFlow Mediator was done by NTT/Hitachi. • Testing of the prototype was conducted using MAWI traffic data. • Objective of this presentation • Introduce implementation of IPFIX/NetFlow mediator to show feasibility of the concept and clarify its importance. • prototype system of IPFIX/NetFlow mediator. • test results of aggregation and storing process.

3. Network monitoring without mediator To monitor the routers traffic matrix, we should collect the entire flow information to one server. - Total traffic: 440Gbps - 200 routers in a network - 220f/s per router (1/1000 sampling) - Total flow rate: 43kf/s Monitoring Server 1 Monitoring Server (Maximum of 10kf/s) 43kf/s Router Router 100Gbps Routers * 200

4. Network monitoring with mediator Mediator stores and aggregates flow information from 20 routers. Monitoring Server 1 Monitoring Server (Maximum of 10kf/s) 43 -> 8.17kf/s (aggregated flows) Mediator Mediator 10 NW domains 10 Mediators Router Router 20 edge routers per domain 100Gbps Routers * 200

5. Aggregation Ratio Dependency • Compare aggregation ratio • 3 different traffic samples • Aggregation Timer: 5s - 180s • Sampling Rate: 1/1 - 1/1024 To utilize the flexibility of aggregation, we need IPFIX mediator.

6. Monitoring Server Monitoring Server aggregated data query request query result Aggregation Device aggregation data IF HDD Query Module aggregated data Stored Traffic Data Aggregation Module Data Store Module traffic information Buffer Module Router IF traffic information Router Mediator Architecture

7. Mediator Prototype Overview • Features • NetFlow ver. 5, ver.9 (IPv4/v6) • Stores flow information in NetFlow format. • Aggregates flow information: • Any-port • DstHost • BGPnexthop • MPLS • System Specification • Implemented in C, Linux OS • NetFlow ver. 5, ver.9 (IPv4/v6) sum IN_BYTES IN_BYTES sum IN_PKTS IN_PKTS key PROTOCOL PROTOCOL discard key IPV4_DST_ADDR INPUT_SNMP IPV4_DST_ADDR key SRC_AS SRC_AS append EXP_IPV4_ADDR append AVE_ACTIVE_TIME

8. Performance Test Result traffic data: MAWI(200602231400.dump)

9. Conclusion • IPFIX/NetFlow mediator is an essential component for realizing scalable & real-time monitoring system in a large-scale network. • Aggregation ratio varies depending on flow numbers and aggregation methods. • We will proceed to study actual deployment of mediators in an operating network environment. • We invite discussions regarding key standardization issues such as exporter information for IPFIX Mediators. • Next step, we will try to refine the IPFIX Mediator draft and draw up this experimental approach.

10. Additional Function? • Modify and create new information elements. • For MPLS NW, append VPN id instead of label value. • For simple 5-tuple flows, append BGP next-hop or AS number. • Handle the exporter information. • To notify the exporter information, we already introduced the new templates in IETF67th. • In some case of exchange the traffic information between the different domain, it intentionally don’t notify exporter information to hide the topology. • In particular, a proxy needs to hide the related exporter information, such as next-hop and ifindex in the flow. • Anonymize private parts of the flow. • For example, DST address or SRC address should be anonymized in some case of situation. • To monitor the traffic trend, it can be anonymized it. It prevent from security violation accident.