Introduction to ESnet Network Operating System (ENOS)

1. Introduction to ESnet Network Operating System (ENOS) SENSE PI Meeting FNAL, February 16th, 2016 Eric Pouyoul (lomax@es.net)

2. Is ESnet really developing Yet Another Network Operating System (YANOS) ? Focus on: • Security • Resource sharing (multi-applications) • design patterns • Integrates ESnet services • Low learning curve and ease of use • ODL, ONOS, other were not available at the time and not with consistent feature set • ESnet traffic is not typical. • Not a controller (does not implement one) • Will leverage any WAN, production quality controller Well, yes, sorry, but we had to ! Co-design with hardware and network

3. ENOS Basic Modules User Management
Local storage 
SSHD support 
Interactive shell
Java and Python support 
Application Scheduler (to be implemented)
Application Life Cycle (partially implemented)
Centralized Logging 
Messaging / Events (in progress)
Web Service Provider (to be implemented)
Topology Service (partially implemented)
Distributed database (not implemented)

4. Optional ENOS Modules ESnet production network topology provider
ESnetTestbed network topology provider
ESnetperfSONAR Testers topology provider
ESnetTestbed Multipoint VPN Service (partial, in progress)
perfSONAR Lookup Service client
ESnet SNMP collector (partial)
OpenDaylight resource provider (partial, in progress)
Inter-ENOS communication (in progress)
Host local network configuration service (in progress)
OVS switches resource provider (in progress)
Linux Virtualization resource provider (partial, in progress)

5. ENOS Modular Architecture REST API, Portal Layer 2 services MultiPoint L2 VPN Layer 3 SDX Netshell security, core services, API, SSHD Controller Drivers ODL, ONOS,… Network Services Drivers Topology, NSI, OSCARS,… Stats, Tests perfSONAR, SNMP, Sampling Database Immediately or eventually consistent VM Driver Linux containers (libvirt)

6. ENOS Building Blocks Java Virtual Machine OSGi Karaf Netshell Bundles, Python ENOS Modules ENOS Services

7. ENOS and Software Technologies • Java Virtual Machine (JVM) for a production-ready environment • ENOS security leverages Java Security Manager • Performance • Python for rapid software development • Useful for experiments, quick prototyping • “The natural language of ENOS” • ENOS integrates Jython, a Python 2.7 interpreter running in the JVM. • OSGi, Karaf for module, application management. • OpenDaylight for controlling SDN devices

8. ENOS Applications An ENOS application runs within the ENOS application framework. It leverages the ENOS services and follows ENOS design patterns. A typical ENOS application architecture will implement some if not all of the following:

9. ENOS Resource Drivers An ENOS Resource Driver is an ENOS application that interfaces to a non-ENOS service. An ENOS driver must convert the third-party, external service API to ENOS data types and API. ENOS ENOS Application Third Party Software or Service Generic API Resource Driver Adaptation

10. ENOS Resource Provider A Resource Provider is an ENOS application that let other ENOS applications access resources. It is responsible for maintaining the state of the resources as well as controlling its access (generally through ACLs). ENOS ENOS Application Resources Generic API An ENOS application can be a resource driver and provider at the same time. Resource Provider Resources ACL

11. ENOS Security Primitives ENOS security leverages the Java Virtual Machine’s security manager in order to provide basic mechanisms that are used by the applications, services and drivers to implement the desired protection to resources: • A system call defining and enforcing privileged and unprivileged execution. • All ENOS threads are associated with an ENOS user. Only privileged users can change the ownership of a thread. • Support for resource access control.

12. Resource Access Control in ENOS

13. Example: ENOS Multipoint VPN Service ENOS DHCP Virtual Machine Management Layer 2 Broadcast MAC Learning Virtualization / Isolation Layer 2 Provisioning Path Computation OSCARS Driver Open Daylight Driver ENOS Topology Service

14. ESnet SDN Testbed SDN Point of Presence (SDN POP) ESnet PE Router (2+)x10GE Planned SDN Testbed node locations Planned SDN Testbed connectivity overlay (using OSCARS circuits) (n)x10GE Testbed Host STAR AMST DENV AOFA LBL WASH CERN ATLA

15. ESnet SDN Testbed Hardware: Corsa Technology DP6440 4 x 100G CFP2 Ports 24x 10G SFP+ Ports DP6430 2 x 100G CFP2 Ports 24x 10G SFP+ Ports 100G 10G DP6420 48 x 10G SFP+ Ports 4 x 40G QSFP+ Ports Open vSwitch + DP6410 24 x 10G SFP+ Ports OpenFlow 1.3 / 1.4 / 1.5+ • FPGA-based SDN switch, with OpenFlow controller interface • Highly scalable both in number of flows and speed • Doing common actions (defined by pipeline) very fast • No broadcast, subset of OpenFlow matches and actions .

16. ESnet SDN POP Logical View ESnet SDN POP Physical View ESnet PE Router (2+)x10GE ESnet PE Router (n)x10GE Testbed Host (2+)x10GE (n)x10GE Services VM

17. ESnet SDN POP Logical View • Software switch and Services VM paired with every hardware switch. • Most flows only pass through hardware switch (green) • Flows requiring special handling go through software switch (blue), possibly to Services VM (red). • Hardware switch provides performance, stability • Software switch and service VM provide flexibility, without compromising reliability ESnet PE Router (2+)x10GE (n)x10GE Services VM

18. Design Patterns at work:Traffic Mirroring. ANL OVS Flow Private OSCARS layer 2 circuit STAR SDN POP LBL CERN Flow Private OSCARS layer 2 circuit Shared OSCARS layer 2 circuits Private OSCARS layer2 circuit CERN SDN POP LBL SDN POP OVS OVS VM

19. Thank You / Questions ? Eric Pouyoul: lomax@es.net / epouyoul@lbl.gov

20. Additional Slides

21. >>> from net.es.netshell.api import TopologyProvider >>> graph = topo.getGraph(TopologyProvider.WeightType.TrafficEngineering) >>> lbl = topo.getNode('lbl-mr2@es.net') >>> amst= topo.getNode('amst-cr5@es.net') >>> from org.jgrapht.alg import DijkstraShortestPath >>> path = DijkstraShortestPath.findPathBetween(graph, lbl, amst) >>> for link in path: ... node = topo.getNodeByLink(link.getId() ... print "Node= " + node.getId() + "\tlinkId= " + link.getId() • Node= urn:ogf:network:es.net:sunn-cr5 linkId= urn:ogf:network:es.net:sunn-cr5:to_lbl-mr2_ip-a:0 • Node= urn:ogf:network:es.net:sacr-cr5 linkId= urn:ogf:network:es.net:sacr-cr5:to_sunn-cr5_ip-a:0 • Node= urn:ogf:network:es.net:denv-cr5 linkId= urn:ogf:network:es.net:denv-cr5:to_sacr-cr5_ip-a:0 • Node= urn:ogf:network:es.net:kans-cr5 linkId= urn:ogf:network:es.net:kans-cr5:to_denv-cr5_ip-a:0 • Node= urn:ogf:network:es.net:chic-cr5 linkId= urn:ogf:network:es.net:chic-cr5:to_kans-cr5_ip-a:0 • Node= urn:ogf:network:es.net:wash-cr5 linkId= urn:ogf:network:es.net:wash-cr5:to_chic-cr5_ip-a:0 • Node= urn:ogf:network:es.net:aofa-cr5 linkId= urn:ogf:network:es.net:aofa-cr5:to_wash-cr5_ip-a:0 • Node= urn:ogf:network:es.net:lond-cr5 linkId= urn:ogf:network:es.net:lond-cr5:to_aofa-cr5_ip-a:0 • Node= urn:ogf:network:es.net:amst-cr5 linkId= urn:ogf:network:es.net:amst-cr5:to_lond-cr5_ip-a:0

22. Lessons Learned and Summary • Java + Python + SSHD is very powerful and yet simple combination. • Aggregating, grooming and normalizing data such as topology simplifies greatly network aware applications. • OSGI/Karaf allows us to painlessly change ODL version, ONOS, other OSGi based controller/applications. • Security is not trivial, especially when using third party software: we needed to turn off security in order to use ODL. • Will be open sourced very soon. A network operating system does really look like a computer operating system. Perhaps Linux itself could be the execution environment for network policies.