Conditions Metadata for TAGs

1. Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010

2. Outline • History of Conditions Metadata • Evolution • Overview of Oracle Schemas for TAGs • TAG DB (Event-wise Metadata) • CATALOG (Dataset Catalogue Metadata) • COMA (COnditions MetadatA) • “Current” COMA Schema • Uses of Conditions Metadata • ELSSI usage • Decoding bit packed event-wise TAG attributes (Trigger) • runBrowserReport • runBrowser • Upload procedure and COMA Loading Status • COMA Browser - Interface Status • Documentation and Links • Plans • Summary Elizabeth Gallas - COMA

3. History of this Conditions Metadata project • 2007: first COMA tables were filled for simulation tests/prototype development • Streaming Test and (FDR) Full Dress Rehearsal exercises (MC) • Run/LB-wise conditions were collected from log files and other sources • And INSERTED into new folders in Conditions database (COOL) • Which formed the prototype for the ATLAS Luminosity calculation • Other Trigger/DAQ COOL folders defined at same time • 2008: COMA tables used by ELSSI prototype (still just MC) • Conditions, particularly trigger configuration, not practical to store event-wise • 2009: start extracting Run/LB-wise information from COOL into COMA to facilitate efficient access to Conditions Metadata for ELSSI for Online Runs • runBrowser developed • Initially private tool: to check data integrity/relationships between COMA tables • We realized this runBrowser would be more generally useful … separate Run-browsing (runBrowser) from Event-browsing (ELSSI) … make runBrowser a stand-alone tool • runBrowserReport emerged • Which will help ELSSI and runBrowser describe underlying COMA info and provides more links to other ATLAS systems • COMA tables expanded to include more conditions in anticipation of use cases Elizabeth Gallas - COMA

4. Evolution in the last year of overall project • TAG DB / ELSSI • evolved to a distributed architecture • Not possible to upload all TAGs at any one Oracle site • ELSSI – also needs to know which TAGs are uploaded at which voluntary sites • An additional relational schema has been added: • ATLAS TAGS CATALOG (Elisabeth Vinek) • Contains processing/upload information needed to deploy distributed TAG services on the grid • The COMA tables use the TAG CATALOG • Upload only Run/LB metadata for Runs in CATALOG • This reduces handling of conditions anomalies, • allowing us to focus on Runs of ‘event analysis interest’ • Steps in Database loading – ideally within hours of reconstruction • TAGs uploaded to Oracle • CATALOG tables updated • COMA tables updated • tho plans now to use Tier-0 DB to advance the COMA loading phase Elizabeth Gallas - COMA

5. Oracle Database: TAG DB and associated metadata tables RUNS 3. COMA tables 1. TAG DB: Event-wise metadata Run LB Event … • TAG DB Event-wise metadata tables • Stores Event-wise attributes: • electron (Et, eta, phi …) • muon (Et, eta, phi …) … and references to RAW, ESD, AOD files • Official data processing chain: RAW  ESD AOD TAG files • Data Catalogue tables • Stores information on file and dataset processing and location • Project name • AMI tag (what processing occurred)… • Sources: AMI (ATLAS Metadata Catalogue, Tier0 … • ‘COMA’ (COnditions MetadatA) tables • Conditions of data taking • Beam conditions • Trigger and DAQ conditions • Magnetic field … • Various sources: Conditions DB, Log files, xml files, email… ORACLE DB 2. DATA Catalogue Tables Elizabeth Gallas - COMA

6. Current Activity • COMA: Loading programs continue to improve/refine • Some areas better resolved than others and still some information needs to be loaded (magnet, fill …) • New: Define Run-wise Aggregate prescales, passthrough • Calculates Run-wise summary: PS, PT, Rerun per triggger • Useful to ELSSI to display active/passive/disabled triggers per Run … So that users do not select triggers in a Run which would yield 0 events • Useful for runBrowser to find the active chains for each Run • Data Quality upload • Loads final LBSUMM DQ assessments when tagged/locked • Includes Detector and Physics DQ assessments • Can envision ways to include virtual flags with this information • Schema and Loading – groundwork laid, interfaces underway • Work with DQ group for verification/refinement Elizabeth Gallas - COMA

7. COMA runBrowser: History and Evolution • A web-based browser was developed for COMA tables to facilitate checking • data integrity and • relationships in the relational tables • March2009: Features of this initial browser were presented … we realized that this kind of browser could be a Run/LB wise component of ELSSI … but also could be a stand-alone browser for Conditions data • http://indico.cern.ch/materialDisplay.py?contribId=1&materialId=slides&confId=54409 • The name of this browser: runBrowser (or COMABroswer) • Not yet released to the collaboration • Not a replacement for runQuery (the web based browser to all online Runs in the Conditions DB) • May2009: TAG group developed first DTD for the GoodRunList XML(GRL): • This xml will be how runBrowser communicates to ELSSI the Run/LB selection. • This DTD has since been taken up by the ATLAS experiment to communicate good Run/LB ranges between subsystems. Elizabeth Gallas - COMA

8. COMA runBrowser: Interface Status • As data has been added to COMA tables • runBrowser has evolved accordingly • Current selection criteria includes: • Data Source (online,simulation), Data Type, Run Number, Duration, Number of LBs, temporal, DAQ configuration, Filename tag (Project Name), Trigger Super Master Key, Trigger LVL1 and/or HLT Prescale Key. • Useful for initial purpose and query development • But initial implementation was not generally well constructed … • October 2009: Ryan Buckingham joined project • Oxford graduate student – 1st year. Move to CERN Aug 1, 2010 • Working on runBrowser • Improved internal communication object oriented structure • introducing further classes - allow criteria to be added in modules • Will facilitate better syntax alignment with runQuery and inclusion of criteria to the GRL xml to pass to ELSSI • Improve look/feel of interface: using php and jQuery widgets Elizabeth Gallas - COMA

9. Documentation and Links • COMA Schema • COMA Tables https://gallas.web.cern.ch/gallas/COMA_Tables.html • COMA developer documentation: https://gallas.web.cern.ch/gallas/Conditions_to_TAGs.html Interface Links NOTE: The interfaces require a grid certificate in you browser NOTE: The interfaces are not in production server locations • runBrowser (2) Link: • runBrowserReport Link: Elizabeth Gallas - COMA

10. Functionality in our Future • Criteria for selecting Runs (some criteria combining COMA/CATALOG Metadata) • Select Online or MC Runs • The TAGs for Online/MC have the same attributes • Select Runs uploaded to TAG DB • These Runs ARE the Runs of analysis interest (the Runs ATLAS chose to reconstruct and reprocess) • These Runs follow the CREM deletion policy • Select Runs with a particular AMI tag • Select Runs with many of the criteria now available in runQuery (but NOT a replacement for runQuery) • Project Name, Magnetic field, Beam conditions, DAQ config, Trigger Master Key (and LVL1/HLT PS Keys) … • Select Runs with one/more particular trigger chains • Moreover those which are active during the Run Elizabeth Gallas - COMA

11. Conclusions • This talk reflects an evolving system • Current information in the system is growing • Adding Conditions Metadata to a relational database • Making that criteria available in a useable interface • We want to insure the Metadata is • complete enough to satisfy use cases while • Reflecting accurately its limitations • Interfaces are being constructed to use selection syntax, criteria, and communication in common use in ATLAS • i.e. runQuery, GoodRunList xml … • This facilitates cross checks with other systems • Continuous process: talking with various experts on insuring • data integrity and completeness Elizabeth Gallas - COMA