1 / 33

NOC Tools Donal O’Cearbhaill HEAnet Ltd.

HEAnet Ltd provides free broadband connectivity to Irish schools, ensuring internet and educational network access. Services include network management, security, email, and monitoring tools like SmokePing, Nagios, Rancid, Cacti, and Netflow. The provisioned system handles services, router configurations, DNS, web hosting, and more.

fbaca
Download Presentation

NOC Tools Donal O’Cearbhaill HEAnet Ltd.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NOC Tools Donal O’Cearbhaill HEAnet Ltd.

  2. Ireland’s National Education and Research Network • Provides Internet services to Irish Universities • 2005 - Broadband for Schools

  3. Broadband for Schools • Free ‘always on’ broadband connectivity to Schools • 3 Year Agreement • Dept of Education/Dept of Communication/TIF • 3,925+ Schools • 7 Access Providers • HEAnet backbone network • Onward connectivity to Internet & Educational Networks • HEAnet Managed Services: Network; Security; E-Mail

  4. Challenges • 4,000 schools • Highly contended links • A lot of satellite connections • SLA/Contract enforcement

  5. Installation Rate

  6. Monitoring/ISP Infrastructure • 28 Debian/Ubuntu servers • 4 Fibrenetix disk arrays • Disk based backup • rsync & application level dumps • Syslog nodes • PostgreSQL database • Aggregation Routers • 7301 • PPPoE • GRE • Border/Services Routers • 6500, 3750

  7. Tools • SmokePing • Nagios • Rancid • Cacti • Netflow

  8. SmokePing • Latency measurement tool • Runs probes in parallel • >3,800 hosts • RRD backend • Reporting • Historical view • Acceptance testing • Tuning • FPing timeouts decreased • Total number of probes reduced • Satellite frequency reduced

  9. Nagios • 4,131 services on 3,905 hosts • Top 5 number of hosts on nagios.org • Populated by SmokePing and memcache • Nagios runs checks serially • >1 hour vs. 15 mins • Nagios populates • sidebar alarms • Schools Up Graph

  10. Rancid • Really Awesome New Cisco confIg Differ • 3,296 Router configs • Maintains history of changes • Mails changes

  11. Cacti • 3,900 hosts • Data gathering • SNMP • External Perl scripts • Graph templating • Database driven • Cricket: 27 mins • Perl • Cacti: <5 mins • Cactid • Custom multithreaded C application

  12. Cacti Weathermap

  13. Interconnects

  14. Netflow • NfSen is a graphical web based front end for the nfdump netflow tools • Query abuse reports • Usage reporting

  15. Reporting Gigabytes downloaded by schools on 22/03/07: 332 Gigabytes uploaded by schools on 22/03/07 : 48 Total MegaBytes downloaded for Digiweb Satellite: 12834 Total MegaBytes uploaded for Digiweb Satellite: 1202 Total MegaBytes downloaded for Digiweb Wireless: 77578 Total MegaBytes uploaded for Digiweb Wireless: 10217 Total MegaBytes downloaded for ESATBT ADSL: 54352 Total MegaBytes uploaded for ESATBT ADSL: 6632 Total MegaBytes downloaded for HSData Wireless: 3047 Total MegaBytes uploaded for HSData Wireless: 575 ….. • Daily Reports • DNS log reporting • Report infected PCs • Top MX lookups • Misconfigurations • Active Directory • Netflow • IPs • Schools usage

  16. Logging • Syslog server per PoP • Servers • Routers • Logcheck • Logfile scanner • IP to school identifier • Mapping IP to school

  17. Server Monitoring • SSH keys • Sharing keys/fingerprints • High overhead • SNMP • Less configurable • Memcache • Local Perl script • Easy to rollout • Load • Disk Space • Monitor Processes

  18. Memcache • Distributed memory caching system • Low overhead • Speed up dynamic database-driven websites by caching data and objects in memory • Developed for LiveJournal • Slashdot • Wikipedia • SourceForge • Schools • Nagios • Maps • Server status

  19. Subversion • Modern replacement for CVS • Provisioning System • Configs • ViewCVS • Checkins get mailed • Schools-noc • Scripts stored on every server • Automatically updated • cron.d

  20. Sidebar • Nagios polled every minute • Populated into memcache • Sidebar alarms • Pubcookie single sign-on

  21. Provisioning System • Services provisioned • CPE router config • Nagios • RADIUS • Cacti • Cisco ACS (TACACS+) • SmokePing • Fortigate (Content filtering) • Maps • DNS • Webhosting

  22. Provisioning System • Text::Template templating system • Data stored in authoritative database • PostgreSQL’s INET type is brilliant! • Perl scripts generate configlets • Added to Subversion • Perl/Shell provisioning agents handle service restarts etc. • Ability to stop all provisioning

  23. Provisioning System Structure

  24. Google Maps

  25. Random things we’ve encountered • Predictable traffic levels • Smokeping, Nagios and Cricket/Cacti take a lot of tuning to monitor our network • Difficult to achieve high bandwidth and high level of reliability in transparent content filter

  26. donal.ocearbhaill@heanet.ie

More Related