Iosif Legrand California Institute of Technology

1. An Agent Based, Dynamic Service System to Monitor, Control and Optimize Distributed Systems March 2006 Iosif Legrand California Institute of Technology March 2006 Iosif Legrand

2. The MonALISA Framework • MonALISA is a Dynamic, Distributed Service System capable to collect any type of information from different systems, to analyze it in near real time and to provide support for automated control decisions and global optimization of workflows in complex grid systems. • The MonALISA system is designed as an ensemble of autonomous multi-threaded, self-describing agent-based subsystems which are registered as dynamic services, and are able to collaborate and cooperate in performing a wide range of monitoring tasks. These agents can analyze and process the information, in a distributed way, and to provide optimization decisions in large scale distributed applications. March 2006 Iosif Legrand

3. MonALISA is A Dynamic, Distributed Service Architecture • The framework is based on a hierarchical structure of loosely coupled agents acting as distributed services which are independent & autonomous entities able to discover themselves and to cooperate using a dynamic set of proxies or self describing protocols. • An agent-based architecture provides the ability to invest the system with increasing degrees of intelligence; to reduce complexity and make global systems manageable in real time. For an effective use of distributed resources, these services provide adaptability and self-organization. March 2006 Iosif Legrand

4. The MonALISA Discovery System & Services Fully Distributed System with no Single Point of Failure Global Services or Clients Clients , HL services repositories Dynamic load balancing Scalability & Replication Security AAA for Clients Proxies AGENTS Distributed System for gathering and Analyzing Information. MonALISA services Distributed Dynamic Discovery- based on a lease Mechanism and REN Network of JINI-LUSs Secure & Public March 2006 Iosif Legrand

5. Monitoring Grid sites, Running Jobs, Network Traffic and Topology JOBS TOPOLOGY ACCOUNTING March 2006 Iosif Legrand

6. Monitoring OSG: Resources, Jobs & Accounting Running Jobs Accounting 42 SITES ~ 4 000 Nodes ( 10 000 CPUs) Thousands of Jobs 60 000 parameters March 2006 Iosif Legrand

7. Monitoring CMS Jobs Running Jobs per Site Type of Jobs Rate for Processing Events per Site Integrated No. of Events March 2006 Iosif Legrand

8. FTP Data Transfer between GRID sites Total FTP Traffic per VO March 2006 Iosif Legrand

9. Monitoring Internet2 backbone Network • Test for a Land Speed Record • ~ 7 Gb/s in a single TCP stream from Geneva to Caltech March 2006 Iosif Legrand

10. The UltraLight Network BNL ESnet IN /OUT March 2006 Iosif Legrand

11. Monitoring The GLORIAD Ring March 2006 Iosif Legrand

12. NETWORKS ROUTERS AS Monitoring Network Topology Latency, Routers March 2006 Iosif Legrand

13. Bandwidth Challenge at SC2005 151 Gbs ~ 500 TB Total in 4h March 2006 Iosif Legrand

14. Available Bandwidth Measurements • one-to-one realtime bandwidth estimation. March 2006 Iosif Legrand

15. Job1 Job Job Job2 Job 31 Job3 Job 32 Monitoring the Execution of Jobs and the Time Evolution SPLIT JOBS LIFELINES for JOBS Summit a Job DAG March 2006 Iosif Legrand

16. App. Monitoring UDP/XDR UDP/XDR UDP/XDR Time;IP;procID MonitoringData MonitoringData MonitoringData parameter1: value parameter2: value ... App. Monitoring Mbps_out:0.52 Status: reading MB_inout: 562.4 ApMon – Application Monitoring Library of APIs (C, C++, Java, Perl. Python) that can be used to send any information to MonALISA services • Flexibility, dynamic configuration, high communication performance dynamic reloading Config Servlet • Automated system monitoring • Accounting information MonALISA hosts APPLICATION MonALISA Service ApMon APPLICATION MonALISA Service ApMon System Monitoring No Lost Packages ApMon configuration generated automatically by a servlet / CGI script load1:0.24 ApMon Config processes: 97 pages_in:83 March 2006 Iosif Legrand

17. End User / Client Agent LISA- Localhost Information Service Agent • Authorization • Service discovery • Local detection of the hardware and software configuration • Complete end-system monitoring: Per-process load, I/O and network throughputs, etc. • End-to-end performance measurements • Will act as an active listener for all events related with the requests generated by its local applications. March 2006 Iosif Legrand

18. Optical Switch Optical Switch Optical Switch ML Agent ML Agent ML Agent MonALISA MonALISA MonALISA MonALISA agents to create on demand on an optical path or tree Discovery & Secure Connection 2 3 ML Demon 1 Time to create a path on demand <1s independent of the location and the number of connections Control and Monitor the switch Runs a ML Demon >ml_path IP1 IP4 “copy file IP4” 4 ML proxy services used in Agent Communication March 2006 Iosif Legrand

19. Monitoring and Controlling Optical Planes Controlling Port power monitoring March 2006 Iosif Legrand

20. Monitoring Optical Switches Agents to Create on Demand an Optical Path March 2006 Iosif Legrand

21. MonALISA MonALISA MonALISA ML Agent ML Agent ML Agent ML Agent ML Agent ML Agent The Functionality of the VINCI System ML proxy services Layer 3 ROUTERS Agent ETHERNET LAN-PHYor WAN-PHY Layer 2 Agent Agent DWDM FIBER Layer 1 Agent Agent Site A Site B Site C March 2006 Iosif Legrand

22. Vertical Integration of Services Real Time Correlations & Feedback between Major Layers is Crucial for Dynamic Load Balancing , Adaptability and Self-Organization MPLS/GMPLS/TL1 Networking Farms & Data Servers Job 31 Job1 Job3 Job2 Job 32 Job Applications Note: Grid Services will have to Interact & Negotiate with Network Services for Network “Resources” Job User “On-Demand”, Dynamicand Self Adapting use of networking resources March 2006 Iosif Legrand

23. Communities using MonALISA Major Communities • OSG • CMS • ALICE • D0 • STAR • VRVS • LGC RUSSIA • SE Europe GRID • APAC Grid • UNAM Grid • ABILENE • ULTRALIGHT • GLORIAD • LHC Net • RoEduNET MonALISA Running 24 X 7 at 250 Sites • Collecting 250,000 parameters in near real-time • Update rate of 25,000 parameter updates per second • Monitoring • 12,000 computers • > 100 WAN Links • Thousands of Grid jobs running con- currently • Demonstrated at: • SC2003 • Telecom 2003 • WSIS 2003 • SC 2004 • I2 2005 • TERENA 2005 • IGrid 2005 • SC 2005 • CENIC 2006 ABILENE - CMS-DC04 - GRID3 VRVS ALICE March 2006 Iosif Legrand

24. The MonALISA Architecture Provides: • Distributed Registration and Discovery for Services and Applications. • Monitoring all aspects of complex systems : • System information for computer nodes and clusters • Network information : WAN and LAN • Monitoring the performance of Applications, Jobs or services • The End User Systems, its performance • Video streaming • Can interact with any other services to provide in near real-time customized information based on monitoring data • Secure, remote administration for services and applications • Agents to supervise applications, trigger alarms, restart or reconfigure them, and to notify other services when certain conditions are detected. • The MonALISA framework is used to develop higher level decision services, implemented as a distributed network of communicating agents, to perform global optimization tasks. • Graphical User Interfaces to visualize complex information March 2006 Iosif Legrand