The GrangeNet NOC

1. The GrangeNet NOC Greg Wickham

2. Contents • Goals • Design Features • Architecture • Implementation • Future • Conclusion

3. Contents • Goals • Design Features • Architecture • Implementation • Future • Conclusion

4. Goals • Central repository where all information about the network is available • Monitoring of the network including availability and utilisation • Access control to ensure that the right people have access to the right data • Leverage existing utilities where practical

5. Contents • Goals • Design Features • Architecture • Implementation • Future • Conclusion

6. Design Features • Easy to use • Intuitive • Web Framework aids rapid development • Modular Construction • Extensible • Security from the ground up • Unified configuration • Mimimise replication of data

7. Contents • Goals • Design Features • Architecture • Implementation • Future • Conclusion

8. Architecture • Log file monitoring • Security server • Database backend • Historical configurations

9. Architecture

10. Architecture Database contains information on: devices; connections; interfaces; as numbers; prefixes; subnets; members db

11. Architecture devices logfiles db Devices connected to the network syslog to the NOC

12. Architecture logwatcher devices logfiles db Logwatcher process monitors log files; messages from devices defined in the db are stored in the db

13. Architecture logwatcher devices logfiles pancho db Logwatcher notices a configuration change; pancho is used to download the config and store in a RCS configs

14. Architecture datacollector utilisation logwatcher devices logfiles pancho db Datacollector periodically examines SNMP counters on device; stores in RRDTool db configs

15. Architecture datacollector tacasip utilisation logwatcher devices logfiles pancho db Tacacs authentication is used for all Cisco devices; no shared enable secret configs

16. Architecture datacollector tacasip utilisation logwatcher devices logfiles pancho db confreport configs Configuration change reports are mailed to the noc-manager nightly

17. Architecture datacollector tacasip utilisation logwatcher devices logfiles pancho db confreport configs www Web server is primary interactive portal

18. Architecture datacollector tacasip utilisation logwatcher devices logfiles pancho db confreport configs weathermap www Weathermap renders graphic of total network utilisation

19. Architecture datacollector tacasip utilisation logwatcher devices logfiles pancho db Grapher renders historical graphs confreport configs grapher weathermap www

20. Architecture datacollector tacasip utilisation logwatcher devices logfiles pancho ‘cookie cutter’ configurations are generated db confreport configs grapher weathermap www Config generator

21. Architecture datacollector tacasip utilisation logwatcher devices logfiles pancho db confreport configs grapher weathermap www Config generator

22. Contents • Goals • Design Features • Architecture • Implementation • Future • Conclusion

23. Leverage Open Source Software

24. The Perl Libraries • Unified configuration format • RCS • Object Orientated HTML • Dynamic Web Framework

25. Unified configuration format • Similar to Apache <NODE apac.router> COORD-RELATIVE edge1.act COORD-SYSTEM polar #polar coords are ( distance, angle ) COORD 130,-45 LABEL TEXT apac RADIUS 20 </NODE>

26. RCS • Based on CPAN RCS • Added support for ‘-p’ option • Diff into array

27. Object Orientated HTML • Much easier to render dynamic HTML • Automatic closure • Inline back patching my $html = new HTML::OO; $html->table()->tr()->td(‘Hello World’); my $a = $html->a( { href => ‘link’ } ); $a->print(“Click here”);

28. Dynamic Web Framework • All web pages are rendered in real time • Concept of Modules and Hierarchy • Integrated cookie based authentication • Integrated with LDAP • Simplifies writing new modules, especially when the new module: • Is required to integrate with existing modules • Needs authentication protection • Can leverage existing library routines

29. The Database • Uses PostgreSQL • Contains information describing the network • Who is connected? • Where are they connected? • What devices are connected? • What interfaces are on the devices? • Asset management

30. The Data Collector • Extracts list of devices to monitor from database • MIBS automatically monitored by defined device classes • Automatically tracks hardware • For interfaces – interface name is used • For others – SNMP index is used (ie: cpu etc)

31. The tacacs server • Tacacs used as authentication protocol • Backended into AARNet LDAP • Stateful protocol - No shared enable password • Accounting (console command logging) • Supports some NAS features

32. The Logwatcher • Monitors logged output from network devices • Logged messages are screen for sanity (bad formatting etc.) • Inserted into the database • Any configuration change triggers a download of the configuration • Store configuration files • stripped of ‘secrets’ • Stored in RCS

33. The Web Server • Uses Apache • Perl embedded interpreter (mod_perl) • mod_ssl / OpenSSL (Security) • Mod_perl • Increases • Apache memory footprint • Flexibility (access internal API) • Performance (db access) • Decreases • CGI script startup time

34. The Configuration Reporter • Every night all registered devices are checked • Are running-config and startup-config the same? • Report any changes made in the last 24 hours • Summary report emailed to noc-manager

35. The weathermapper • Goal – At a glance network status • Configuration file defines the weathermap • Nodes: Defines nodes and connections • Map: Defines nodes to draw • Automatically generates: • Full map (all data rendered as requested) • Quick mapping • Background pre-rendered • Foreground drawn over background • Meta data for ISMAP clicking

36. The weathermapper • Goal – At a glance network status • Configuration file defines the weathermap • Nodes: Defines nodes and connections • Map: Defines nodes to draw • Automatically generates: • Full map (all data rendered as requested) • Quick mapping • Background pre-rendered • Foreground drawn over background • Meta data for ISMAP clicking

37. The weathermapper • Embedded Nagios status

38. The weathermapper • Throughput on link • Percentage utilisation indicated by colour

39. The weathermapper • Legend for colurs / percentage utilisation

40. The weathermapper • Links down

41. The weathermapper • Connection status highlight (In progress)

42. The Support Cast • Suite of scripts that: • Collect SNMP data • Check interface names against db • Analyse Nagios status (scraping)

43. The Result • Live demo http://noc.grangenet.net GrangeNet Survey

44. Contents • Goals • Design Features • Architecture • Implementation • Future • Conclusion

45. The Future • Nagios? – Where does it fit • Automatic config generation (from db) • Many similar queries to the same device aren’t efficient • More reporting • Link statistics • Bandwidth utilisation • Shim layer • Scraping currently used • Improved efficiency if APIs are well defined • Ie: Nagios

46. Contents • Goals • Design Features • Architecture • Applications • Implementation • Future • Conclusion

47. Conclusion • Perl – Rapid application development • Proven to be very useful • Centralised data • Describes the entire network • The one source of information

48. Questions? greg.wickham@grangenet.net http://noc.grangenet.net