A universal framework for data validation
This presentation is the property of its rightful owner.
Sponsored Links
1 / 22

A Universal Framework for Data Validation PowerPoint PPT Presentation


  • 164 Views
  • Uploaded on
  • Presentation posted in: General

A Universal Framework for Data Validation. Giovanni Organtini INFN-Sez. di Roma. Data Validation. Data Validation ( DV ) Fast analysis of data coming from a DAQ system to evaluate data usability Each atomic analysis program is called task fast, light and single-objective

Download Presentation

A Universal Framework for Data Validation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


A universal framework for data validation

A Universal Framework for Data Validation

Giovanni Organtini

INFN-Sez. di Roma


Data validation

Data Validation

  • Data Validation (DV)

    • Fast analysis of data coming from a DAQ system to evaluate data usability

    • Each atomic analysis program is called task

      • fast, light and single-objective

  • Data Validation must be

    • Able to evolve on a short timescale

    • Flexible to manage multiple and different validation tasks

    • Robust to allow many different users to cooperate without interference

ACAT03-Giovanni Organtini


Dv framework concepts

DV Framework Concepts

  • Sensors: elements that detect events such as an EOR or a Warning signal issued by analysis tasks

  • Events: any Sensor outcome

  • Mediator: a process that collects Events from Sensors and dispatches them to Solutions

  • Solutions: objects that recognize Events and act according to their content

ACAT03-Giovanni Organtini


Framework architecture

Framework architecture

ACAT03-Giovanni Organtini


Framework architecture1

Framework architecture

Sensors

ACAT03-Giovanni Organtini


Framework architecture2

Framework architecture

Notify Event to

Mediator

ACAT03-Giovanni Organtini


The sensors

The Sensors

  • AlarmSensor: parses HTML-formatted files and detects comments of the form

    <!–- SensorHTML

    Creates events according to comment content

  • EORSensor: parses a text file (runlist) and detects updates. A line is added to runlist at each EOR

ACAT03-Giovanni Organtini


Framework architecture3

Framework architecture

Dispatch Event to

Solutions

ACAT03-Giovanni Organtini


Framework architecture4

Framework architecture

Solutions

ACAT03-Giovanni Organtini


The solutions

The Solutions

  • Program: launch one or more programs in a given directory (repair known problems)

  • E-mail: send an e-mail with arbitrary text to a list of recipient

  • Window: show a popup window on a given terminal with arbitrary HTML-formatted text

ACAT03-Giovanni Organtini


A more formal view

A more formal view

  • Implemented in Java

    • portable

    • distributed (RMI-server created to allow Sensors to run on remote processors)

ACAT03-Giovanni Organtini


How it works

How it works

  • Sensors detect changes in files

    • an EOR-Sensor can detect an EOR looking at the content of a given file containing the runlist

    • An AlarmSensor can parse output files produced by analysis tasks to detect potential problems

  • The Mediator collects Events and notify them to all the existing Solutions

  • The Solutions choose to activate themselves according to Event content

ACAT03-Giovanni Organtini


Eor sensor

EOR-Sensor

  • The EOR-Sensor generate an Event as soon as a new line appear in the runlist

  • runlist formatted as follows

    <commands/beam/> --date=20031111 –ebeam=50 –beam=e

ACAT03-Giovanni Organtini


Eor sensor1

EOR-Sensor

  • The EOR-Sensor generate an Event as soon as a new line appear in the runlist

  • runlist formatted as follows

    <commands/beam/> --date=20031111 –ebeam=50 –beam=e

directory containing

analysis tasks for

this application

ACAT03-Giovanni Organtini


Eor sensor2

EOR-Sensor

  • The EOR-Sensor generate an Event as soon as a new line appear in the runlist

  • runlist formatted as follows

    <commands/beam/> --date=20031111 –ebeam=50 –beam=e

directory containing

analysis tasks for

this application

parameters to be passed to

analysis tasks (arbitrary)

ACAT03-Giovanni Organtini


Eor sensor3

EOR-Sensor

  • The EOR-Sensor generate an Event as soon as a new line appear in the runlist

  • runlist formatted as follows

    <commands/beam/> --date=20031111 –ebeam=50 –beam=e

  • Program Solution activated

directory containing

analysis tasks for

this application

parameters to be passed to

analysis tasks (arbitrary)

ACAT03-Giovanni Organtini


Analysis tasks

Analysis tasks

  • Analysis tasks can be written in ANY language, provided they can accept arguments on the command line

    • Examples: C, C++, Perl, shell scripts, etc.

  • They must provide the output in HTML-formatted files to be located in a given directory

    • output can be eithertext and/or images

    • the result of any single task could be browsed from everywhere using any browser

    • a process will automatically generate the list of output files to be shown to the shifter

ACAT03-Giovanni Organtini


Alarm sensors

Alarm Sensors

  • They look for HTML-comments in output files and generate events according to the comment content

    • comments not shown by browsers

    • must be compliant with a set of rules

    • actions may be combined (i.e. can send e-mails and notify a message to the shifter)

ACAT03-Giovanni Organtini


Example

Example

<!– SensorHTML

winFlag=false

emailFlag=true

commandFlag=false

<emailBody>

[email protected]

senderName=Giovanni Organtini - ped analyzer

[email protected],[email protected]

smtpServer=localhost

subject=Problem with pedestals in test beam

<message>

A problem has occurred during the analysis of the

pedestal. May be you want to run to the detector to

investigate…

</message>

</emailBody>

..-->

ACAT03-Giovanni Organtini


Solution activation

Solution activation

  • The e-mail Solution is activated

    • sends an e-mail to Coordinator and Task responsible

    • the e-mail body is arbitrary and may contain links and other HTML-tags

  • Other informative message can be written on the output file containing the comment

    • output file may contain

      • text

      • images

      • links

ACAT03-Giovanni Organtini


Applications

Applications

  • CMS Crystal Calorimeter Construction in Roma

    • Crystals measured by automatic machine

    • Results analyzed by validation tasks

    • Problems notified to construction supervisor via e-mail

    • Some known problems recovered via program Solution

  • CMS Crystal Calorimeter Test-Beam at CERN

    • Calorimeter modules exposed to e, p beams

    • Data collected and analyzed by validation tasks

    • Problems notified to experts via e-mail and to shifters via popup windows

ACAT03-Giovanni Organtini


Conclusion

Conclusion

  • A Data Validation Framework has been developed

    • simple to manage and use

    • universal

    • flexible

    • robust

    • configurable

    • no requirements on analysis tasks

    • no library to link or file to include

  • Package available soon (2004) for download on http://www.roma1.infn.it/people/organtini

ACAT03-Giovanni Organtini


  • Login