CMS: XDAQ BTeV: RTES, GME, ARMORs EPICS SMI++

1. FAIR Controls CMS: XDAQ BTeV: RTES, GME, ARMORs EPICS SMI++ H.G.Essel, J.Adamczewski, B.Kolb, M.Stockmeier Hans G. Essel DAQ Control

2. CMS XDAQ • XDAQ is a framework targeted at data processing clusters • Can be used for general purpose applications • Has its origins in the I2O (Intelligent IO) specification • The programming environment is designed as an executive • A program that runs on every host • User applications are C++ programmed plug-ins • Plug-ins are dynamically downloaded into the executives • The executive provides functionality for • Memory management • Systems programming queues, tasks, semaphores, timers • Communication Asynchronous peer-to-peer communication model Incoming events (data, signals, …) are demultiplexed to callback functions of application components • Services for configuration, control and monitoring • Direct hardware access and manipulation services • Persistency services Hans G. Essel DAQ Control

3. XDAQ event driven communication • Dynamically loaded application modules (from URL, from file) • Inbound/Outbound queue (pass frame pointers, zero-copy) • Homogeneous frame format Readout component Generates a DMA completion event Computer Executive framework Demultiplexes incoming events to listener application component foo( ) Application component Implements callback function Peer transport Receives messages from network Hans G. Essel DAQ Control

4. Application component device Processing nodeIOP Controller node host Homogeneous communication frameSend for local, remote, host single addressing scheme (Tid) Application framework XDAQ: I2O peer operation for clusters Messaging Layer Messaging Layer ‚ ‡ ˆ Peer Transport Agent Peer Transport Agent † I2O Message Frames  Executive Executive ƒ „ … Application ‰ Application Peer Transport Hans G. Essel DAQ Control

5. XDAQWin client Configuration tree XML based configuration of a XDAQ cluster Daqlet window Daqlets are Java applets that can be used to customize the configuration, control and monitoring of all components in the configuration tree Hans G. Essel DAQ Control

6. BTeV: RTES deliverables A hierarchical fault management system and toolkit: • Model Integrated Computing • GME (Generic Modeling Environment) system modeling tools • and application specific “graphic languages” for modeling system configuration, messaging, fault behaviors, user interface, etc. • ARMORs (Adaptive, Reconfigurable, and Mobile Objects for Reliability) • Robust framework for detection and reaction to faults in processes • VLAs (Very Lightweight Agents for limited resource environments) • To monitor/mitigate at every level • DSP, Supervisory nodes, Linux farm, etc. Hans G. Essel DAQ Control

7. Global Operations Manager Logical Control Network Logical Data Net Logical Data Net Global Fault Manager Region Fault Mgr RTES structure Modeling Analysis Resource Reconfigure Performance Diagnosability Reliability Synthesis Design and Analysis Fault Behavior Feedback Algorithms Synthesis Runtime Region Operations Mgr ExperimentControl Interface L2,3/CISC/RISC L1/DSP Soft Real Time Hard Hans G. Essel DAQ Control

8. GME: data type modeling • Modeling of Data Types and Structures • Configure marshalling-demarshalling interfaces for communication Hans G. Essel DAQ Control

9. FMML Model – Behavior Aspect Translator ARMOR ARMOR Microkernel Switch(cur_state) case NOMINAL: I f (time<100) { next_state = FAULT; } Break; case FAULT if () { next_state = NOMINAL; } break; Fault Tolerant Custom Element Communication Custom Element class armorcallback0:public Callback { public:ack0(ControlsCection *cc, void *p) : CallbackFaultInjectTererbose>(cc, p) { } void invoke(FaultInjecerbose* msg) { printf("Callback. Recievede dtml_rcver_LocalArmor_ct *Lo; mc_message_ct *pmc = new m_ct; mc_bundle_ct *bundlepmc->ple(); pmc->assign_name(); bundle=pmc->push_bundle();mc); } }; RTES: GME fault mitigation modeling language (2) • Model translator generates fault-tolerant strategies and communication flow strategy from FMML models • Strategies are plugged into ARMOR infrastructure as ARMOR elements • ARMOR infrastructure uses these custom elements to provide customized fault-tolerant protection to the application Hans G. Essel DAQ Control

10. ARMOR • Adaptive Reconfigurable Mobile Objects of Reliability: • Multithreaded processes composed of replaceable building blocks • Provide error detection and recovery services to user applications • Hierarchy of ARMOR processes form runtime environment: • System management, error detection, and error recovery services distributed across ARMOR processes. • ARMOR Runtime environment is itself self checking. • 3-tiered ARMOR support of user application • Completely transparent and external support • Enhancement of standard libraries • Instrumentation with ARMOR API • ARMOR processes designed to be reconfigurable: • Internal architecture structured around event-driven modules called elements. • Elements provide functionality of the runtime environment, error-detection capabilities, and recovery policies. • Deployed ARMOR processes contain only elements necessary for required error detection and recovery services. • ARMOR processes resilient to errors by leveraging multiple detection and recovery mechanisms: • Internal self-checking mechanisms to prevent failures from occurring and to limit error propagation. • State protected through checkpointing. • Detection and recovery of errors. • ARMOR runtime environment fault-tolerant and scalable: • 1-node, 2-node, and N-node configurations. Hans G. Essel DAQ Control

11. ExecARMOR AppProcess Execution ARMOR Oversees application process(e.g. the various Trigger Supervisor/Monitors) Daemons Detect ARMOR crash and hang failures Daemon network Heartbeat ARMOR Detects and recovers FTM failures Fault Tolerant Manager Highest ranking manager in the system Daemon Daemon ARMOR processes Provide a hierarchy of error detection and recovery.ARMORS are protected through checkpointingand internal self-checking. Fault TolerantManager (FTM) HeartbeatARMOR ARMOR system: basic configuration Hans G. Essel DAQ Control

12. EPICS overview EPICS is a set of software components and tools to develop control systems. The basic components are: OPI (clients) • Operator Interface. This is a UNIX or Windows based workstation which can run various EPICS tools (MEDM, ALH, OracleArchiver). IOC (server) • Input Output Controller. This can be VME/VXI based chassis containing a Motorola 68xxx processor, various I/O modules, and VME modules that provide access to other I/O buses such as GPIB, CANbus. PV (Process variables) • Named objects located in data base records (data and functions) of IOCs LAN (communication) • Local area network. This is the communication network which allows the IOCs and OPIs to communicate. EPICS provides a software component, Channel Access, which provides network transparent communication between a Channel Access client and an arbitrary number of Channel Access servers. Hans G. Essel DAQ Control

13. Hierarchy in a flat system IOC tasks • IOCs • One IOC per standard CPU (Linux, Lynx, VxWorks) • clients • on Linux, (Windows) • Agents • Segment IOCs beeing also clients Name space architecture! IOC IOC tasks Client IOC tasks IOC tasks IOC Hans G. Essel DAQ Control

14. Task command threadworking threadmessage thread Local communication (node) Node • Commands handled by threads • Execution maybe in working thread • Message thread maybe not needed commands intertask Task statussegment IOC memory Task messages Hans G. Essel DAQ Control

15. Screen shot FOPI Hans G. Essel DAQ Control

16. SMI++ and State Management Language SML • Classes and Objects • Allow the decomposition of a complex system into smaller manageable entities • Finite State Machines • Allow the modeling of the behavior of each entity and of the interaction between entities in terms of STATES and ACTIONS • Rule-based reasoning • Allow Automation and Error Recovery • SMI++ Objects can be: • Abstract (e.g. a Run or the DCS) • Concrete (e.g. a power supply or a temp. sensor) • Concrete objects are implemented externally either in "C", in C++, or in PVSS (ctrl scripts) • Logically related objects can be grouped inside "SMI domains" representing a given sub-system • Finite State Logic • Objects are described as FSMstheir main attribute is a STATE • Parallelism • Actions can be sent in parallel to several objects. Tests on the state of objects can block if the objects are still “transiting” • Asynchronous Rules • Actions can be triggered by logical conditions on the state of other objects Hans G. Essel DAQ Control

17. SMI Domain Obj SMI Domain Obj Obj Obj Obj Obj Obj Obj Proxy Proxy Proxy SMI++ Run-time Environment • Device Level: Proxies • C, C++, PVSS ctrl scripts • drive the hardware: • deduceState • handleCommands • Abstract Levels: Domains • Internal objects • Implement the logical model • Dedicated language • User Interfaces • For User Interaction • Communication DIM Hardware Devices Hans G. Essel DAQ Control

18. Kind of conclusion • RTES: Very big and powerful. Not simply available! • Big collaboration • Fully modelled and simulated using GME • ARMORs for maximum fault tolerance and control • XDAQ: Much smaller. Installed at GSI. • Dynamic configurations (XML) • Fault tolerance? • EPICS: From accelerator controls community. Installed at GSI • Maybe best known • No fault tolerance • Not very dynamic • SMI++ DIM: State machines and light weight communication • Used in JCOB (CERN) Hans G. Essel DAQ Control