a universal framework for data validation
Download
Skip this Video
Download Presentation
A Universal Framework for Data Validation

Loading in 2 Seconds...

play fullscreen
1 / 22

A Universal Framework for Data Validation - PowerPoint PPT Presentation


  • 206 Views
  • Uploaded on

A Universal Framework for Data Validation. Giovanni Organtini INFN-Sez. di Roma. Data Validation. Data Validation ( DV ) Fast analysis of data coming from a DAQ system to evaluate data usability Each atomic analysis program is called task fast, light and single-objective

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'A Universal Framework for Data Validation' - abdalla


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
a universal framework for data validation

A Universal Framework for Data Validation

Giovanni Organtini

INFN-Sez. di Roma

data validation
Data Validation
  • Data Validation (DV)
    • Fast analysis of data coming from a DAQ system to evaluate data usability
    • Each atomic analysis program is called task
      • fast, light and single-objective
  • Data Validation must be
    • Able to evolve on a short timescale
    • Flexible to manage multiple and different validation tasks
    • Robust to allow many different users to cooperate without interference

ACAT03-Giovanni Organtini

dv framework concepts
DV Framework Concepts
  • Sensors: elements that detect events such as an EOR or a Warning signal issued by analysis tasks
  • Events: any Sensor outcome
  • Mediator: a process that collects Events from Sensors and dispatches them to Solutions
  • Solutions: objects that recognize Events and act according to their content

ACAT03-Giovanni Organtini

framework architecture
Framework architecture

ACAT03-Giovanni Organtini

framework architecture1
Framework architecture

Sensors

ACAT03-Giovanni Organtini

framework architecture2
Framework architecture

Notify Event to

Mediator

ACAT03-Giovanni Organtini

the sensors
The Sensors
  • AlarmSensor: parses HTML-formatted files and detects comments of the form

Creates events according to comment content

  • EORSensor: parses a text file (runlist) and detects updates. A line is added to runlist at each EOR

ACAT03-Giovanni Organtini

framework architecture3
Framework architecture

Dispatch Event to

Solutions

ACAT03-Giovanni Organtini

framework architecture4
Framework architecture

Solutions

ACAT03-Giovanni Organtini

the solutions
The Solutions
  • Program: launch one or more programs in a given directory (repair known problems)
  • E-mail: send an e-mail with arbitrary text to a list of recipient
  • Window: show a popup window on a given terminal with arbitrary HTML-formatted text

ACAT03-Giovanni Organtini

a more formal view
A more formal view
  • Implemented in Java
    • portable
    • distributed (RMI-server created to allow Sensors to run on remote processors)

ACAT03-Giovanni Organtini

how it works
How it works
  • Sensors detect changes in files
    • an EOR-Sensor can detect an EOR looking at the content of a given file containing the runlist
    • An AlarmSensor can parse output files produced by analysis tasks to detect potential problems
  • The Mediator collects Events and notify them to all the existing Solutions
  • The Solutions choose to activate themselves according to Event content

ACAT03-Giovanni Organtini

eor sensor
EOR-Sensor
  • The EOR-Sensor generate an Event as soon as a new line appear in the runlist
  • runlist formatted as follows

--date=20031111 –ebeam=50 –beam=e

ACAT03-Giovanni Organtini

eor sensor1
EOR-Sensor
  • The EOR-Sensor generate an Event as soon as a new line appear in the runlist
  • runlist formatted as follows

--date=20031111 –ebeam=50 –beam=e

directory containing

analysis tasks for

this application

ACAT03-Giovanni Organtini

eor sensor2
EOR-Sensor
  • The EOR-Sensor generate an Event as soon as a new line appear in the runlist
  • runlist formatted as follows

--date=20031111 –ebeam=50 –beam=e

directory containing

analysis tasks for

this application

parameters to be passed to

analysis tasks (arbitrary)

ACAT03-Giovanni Organtini

eor sensor3
EOR-Sensor
  • The EOR-Sensor generate an Event as soon as a new line appear in the runlist
  • runlist formatted as follows

--date=20031111 –ebeam=50 –beam=e

  • Program Solution activated

directory containing

analysis tasks for

this application

parameters to be passed to

analysis tasks (arbitrary)

ACAT03-Giovanni Organtini

analysis tasks
Analysis tasks
  • Analysis tasks can be written in ANY language, provided they can accept arguments on the command line
    • Examples: C, C++, Perl, shell scripts, etc.
  • They must provide the output in HTML-formatted files to be located in a given directory
    • output can be eithertext and/or images
    • the result of any single task could be browsed from everywhere using any browser
    • a process will automatically generate the list of output files to be shown to the shifter

ACAT03-Giovanni Organtini

alarm sensors
Alarm Sensors
  • They look for HTML-comments in output files and generate events according to the comment content
    • comments not shown by browsers
    • must be compliant with a set of rules
    • actions may be combined (i.e. can send e-mails and notify a message to the shifter)

ACAT03-Giovanni Organtini

example
Example

winFlag=false

emailFlag=true

commandFlag=false

[email protected]

senderName=Giovanni Organtini - ped analyzer

[email protected],[email protected]

smtpServer=localhost

subject=Problem with pedestals in test beam

A problem has occurred during the analysis of the

pedestal. May be you want to run to the detector to

investigate…

..-->

ACAT03-Giovanni Organtini

solution activation
Solution activation
  • The e-mail Solution is activated
    • sends an e-mail to Coordinator and Task responsible
    • the e-mail body is arbitrary and may contain links and other HTML-tags
  • Other informative message can be written on the output file containing the comment
    • output file may contain
      • text
      • images
      • links

ACAT03-Giovanni Organtini

applications
Applications
  • CMS Crystal Calorimeter Construction in Roma
    • Crystals measured by automatic machine
    • Results analyzed by validation tasks
    • Problems notified to construction supervisor via e-mail
    • Some known problems recovered via program Solution
  • CMS Crystal Calorimeter Test-Beam at CERN
    • Calorimeter modules exposed to e, p beams
    • Data collected and analyzed by validation tasks
    • Problems notified to experts via e-mail and to shifters via popup windows

ACAT03-Giovanni Organtini

conclusion
Conclusion
  • A Data Validation Framework has been developed
    • simple to manage and use
    • universal
    • flexible
    • robust
    • configurable
    • no requirements on analysis tasks
    • no library to link or file to include
  • Package available soon (2004) for download on http://www.roma1.infn.it/people/organtini

ACAT03-Giovanni Organtini

ad