The Message Reporting System of the ATLAS DAQ

1. The Message Reporting System of the ATLAS DAQ 10th ICATPP Conference on Astroparticle, Particle, Space Physics, Detectors and Medical Physics Applications Villa Olmo,Como,Italy 8-12 October 2007 M.Caprini IFIN-HH Bucharest, Romania I.Fedorko CERN Geneva, Switzerland S.Kolos UCI California, USA The Data Acquisition System of ATLAS ATLAS is multipurpose experiment detecting the outcome of proton collisions provided by  the Large Hadron Collider (LHC). The ATLAS Data Acquisition System (DAQ) system is composed of ~ 25000 of applications running on more then 2600 interconnected computers. The Online Software as a part of the DAQ is responsible for configuring, controlling and monitoring the data taking system; it provides services to DAQ and detectors system, is responsible for synchronization and supervision of processes. Many applications have to provide information about their status or report error conditions. A stable and scalable distributed message transport mechanism is needed. The Message Reporting System facility, as a part of Online Software, is providing  the means to exchanging unique  information and error messages  across different DAQ applications. • Overview of the MRS • The MRS system is responsible for transport, distribution and filtering of messages. • It has been designed following the client-server paradigm. • Provides three interfaces: • Sender API  for application whishing report messages • Receiver API  application subscribing for requested messages • Commander API  DAQ expert can change some running server settings • The server functionality has been distributed over a publicly accessible MRS server process and a variable number of private MRS servers, called workers, which are performing the actual work of receiving each a sub-set of the messages and to forwarding them to the subscribers. The inter-process communication is achieved using the CORBA technology. The MRS Message flow and the Sender API The MRS stream implements the MRS sender functionality. Message can be created by user or is built from issue provided by from Error Reporting System (ATLAS standard for error reporting) How the message is built and transported: Template: mrsStream << messageID << severity << text<< optional qualifier << parameter<<optional parameter << end of message Example of user message: mout << M1 << MRS_ERROR << MRS_QUALIF("RUNCONTROL") << MRS_TEXT("Long sender text") << MRS_OPT_PARAM<int>("ErrorNumber", "ERROR IS: %d ", 12) << MRS_PARAM<const char*>("param_no", "notInDB") << ENDM; Receiver API An application wishing to receive messages must subscribe to the public MRS server with the help of the MRS receiver interface. Subscription includes the request to receive messages, implemented as a callback mechanism  subscription criteria: any logical expression built from message ID, severity and qualifiers, e.g. (ERROR|WARNING)&(^RC::NotConfig*) Automatically added qualifier by sender API: date, time, sender hostname and application name server processing If message is in DB, add corresponding text and qualifier e.g. text End of Run The Public and Private servers implementation The server functionality was split to publicly accessible MRS server with scalable number of private servers, called workers, which process messages. Having been started up the private MRS servers join the public MRS server sending a handle (self handle) which can be used later to access the methods of the private MRS servers. A user application wishing to report messages has to connect to the public MRS server via the MRS sender API and get handle to a particular private MRS server, sending the messages directly to the private MRS server. The receiver sends subscribe and unsubscribe request to the public MRS server which then transmits this (un)subscription to all private servers. All private MRS servers report any corresponding message directly to the MRS receiver application. Filtering criteria build by the same rules as subscription expression used in receiver Running over list of subscribed receiver Final message received by subscribing application: message1ERROR Long sender text RUNCONTROLEnd of Run16084 lnxatd53.cern.ch 2007/Oct/01 16:42:47 MRS::TestSender param_no notInDB ErrorNumber ERROR IS: 5 Message receiver in the DAQ GUI The DAQ expert can initiate set of command via the MRS command API, e.g. command can adjust the filter criteria for messages on their way through the private MRS servers. Filter out messages matching filter criteria Simplified class diagram of the MRS system The MRS performance The MRS functionality is regularly tested during detector and trigger commissioning and proved that the MRS is able to fulfill expected functionality and reliability. Also the local configurable TDAQ testing setups are used to perform the test and measurements. Setup A: public server + 1 private server hosted on the same machine (4 core Intel Xeon CPU 3GHz, 2GB RAM), no receiver, applications sending messages at maximum rate  for one sending application average message processing rate at the server is ~ 9kHz, for more parallel applications the saturated level is ~17kHz. Receiver subscription criteria for GUI message panel Example of chained messages Setup B: 1,2 and 3 private servers join public space, running separately on 2 core Intel Xeon CPU 2GHz, 500MB RAM. Applications are sending messages at maximum rate, running on PCs of the same type as the private servers. All used receivers (1 core Intel Xeon CPU 2Ghz, 500MB RAM) print messages to file. Message severity Source application Message name Good scalability of message processing rate by using the private servers!