Event data history
Download
1 / 27

- PowerPoint PPT Presentation


  • 245 Views
  • Updated On :

Event Data History. David Adams BNL Atlas Software Week December 2001. Contents. Definitions Event Processing Use cases Requirements Design issues Status Future. Definitions. Data object Unit of data transfer to and from data store Event data object (EDO)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about '' - paul2


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Event data history l.jpg

Event Data History

David Adams

BNL

Atlas Software Week

December 2001


Contents l.jpg
Contents

  • Definitions

  • Event Processing

  • Use cases

  • Requirements

  • Design issues

  • Status

  • Future

Event Data History David Adams BNL


Definitions l.jpg
Definitions

  • Data object

    • Unit of data transfer to and from data store

  • Event data object (EDO)

    • Data object associated with a particular event

    • Restrict to the highest level (not subobjects)

  • Event data contained object

    • Objects contained in EDO’s

    • E.g. cluster, track or electron

    • Only found in and accessed through an EDO

Event Data History David Adams BNL


Definitions cont l.jpg
Definitions (cont)

  • Event

    • A collection of EDO’s (and their histories) associated with one beam crossing

    • Plus virtual EDO’s (histories without data)

    • Not necessarily all such data anywhere

    • Depends on scope; for example:

      • All EDO’s in a file

      • All EDO’s accessible at a site

      • All EDO’s registered in central store

      • Someone’s view of the data

      • All EDO’s visible to an algorithm

Event Data History David Adams BNL


Definitions cont5 l.jpg
Definitions (cont)

  • Algorithm

    • Event data is processed by running a series of algorithms

    • Input to an algorithm are EDO’s and possibly non-event data such as geometry or calibration

    • Output is typically one EDO (can be more)

    • Characterized by type, version and a collection of run-time parameters

    • Similar to the Gaudi Algorithm class except does not include specification of input data

      • Gaudi definition might depend on event

Event Data History David Adams BNL


Definitions cont6 l.jpg
Definitions (cont)

  • Parent EDO

    • Each EDO is constructed by an algorithm from a well-specified collection of input EDO’s

    • These input EDO’s are the parents

    • Each EDO has a well defined ancestry

      • Parents

      • Parents of parents

      • And so on

Event Data History David Adams BNL


Definitions cont7 l.jpg
Definitions (cont)

  • Replicated data

    • Copy a collection of EDO’s

    • Levels:

      • File replication

      • EDO replication (more difficult)

  • Regenerated data

    • Data reconstructed by providing input and then running a collection of algorithms equivalent to those used in the original data generation

    • Regenerate at EDO level

Event Data History David Adams BNL


Definitions cont8 l.jpg
Definitions (cont)

  • Event data history

    • Includes

      • Input data

      • Algorithms used to produce the data

      • Run-time environment

      • Nonessential information (time stamp…)

    • Provide history for each EDO

      • Combine with ancestor histories to recover the full production chain

      • Share information to save space

Event Data History David Adams BNL


Event processing l.jpg
Event Processing

Raw

Data

Track

Clusters

Tracks 1

Tracks 2

EDO + history

non-const

const

Cluster

Find 1

Refit

Algorithm

Find 2

Tracks 3

Event Data History David Adams BNL


Use cases l.jpg
Use cases

  • 1. Check history

    • User wishes to discover the track fitting algorithm used for an electron

    • Electron consists of a track and an EM cluster

      • The track and EM cluster EDO’s are parents of the electron EDO

    • The history for the electron EDO is used to find the history for the parent track EDO

    • The specification of the track fitting algorithm is obtained from the history of the track EDO

Event Data History David Adams BNL


Use cases cont l.jpg
Use cases (cont)

  • 2. Select on history

    • User has a collection of track EDO’s and wishes to find those generated with particular algorithm characteristics

    • User iterates over the associated EDO histories

      • Extract the algorithm data for each

      • Save histories with matching algorithm characteristics

    • User fetches the EDO’s associated with the saved histories

Event Data History David Adams BNL


Use cases cont12 l.jpg
Use cases (cont)

  • 3. Virtual (regenerated) data

    • User wishes to reproduce refit tracks which have been deleted (or never created)

    • Original (parent) track EDO is present

    • Refit track history is present and provides

      • fitting algorithm (including runtime parameters)

      • parent track EDO

    • The fitting algorithm is (re)run

      • Original tracks are used as input

      • Regenerated refit tracks as output

Event Data History David Adams BNL


Use cases cont13 l.jpg
Use cases (cont)

  • 4. Replicated data

    • Track EDO’s for interesting events are replicated in a file at a remote site

    • Later a user desires to refit these tracks:

      • Parent clusters are replicated in a separate file

      • Job is run with both files as input

      • The cluster EDO’s in the second file are recognized as parents of the track EDO’s in the first file

      • These clusters are used to refit the tracks

Event Data History David Adams BNL


Use cases cont14 l.jpg
Use cases (cont)

  • 5. History creation in Gaudi

    • The history data is extracted while running in the Gaudi framework

      • A historian is created at the beginning of the job and it extracts job level history from the OS and Gaudi

      • The historian extracts algorithm-specific history from each algorithm

      • Each time an EDO is created, the historian uses the algorithm and input data (EDO’s and global) to construct history for that EDO

      • The job and algorithm histories can be shared

Event Data History David Adams BNL


Requirements l.jpg
Requirements

  • 1. History includes essential information:

    • Parent EDO’s

    • Relevant global data (calib, alignment, …)

    • Algorithm

      • Type, version and run-time parameters

    • Release version

    • Run time environment

      • OS, OS and shared lib versions

  • 2. Above must be sufficient to reproduce EDO

Event Data History David Adams BNL


Requirements cont l.jpg
Requirements (cont)

  • 3. History includes nonessential information:

    • Event identifier

    • Time stamp

    • Computer identifier

    • Job identifier

    • CPU time consumed

    • Algorithm return status

    • Checksum to verify data

Event Data History David Adams BNL


Requirements cont17 l.jpg
Requirements (cont)

  • 4. History can exist even if its EDO is deleted

  • 5. EDO index

    • There must be a way to label (index) an EDO so references to parent EDO’s can be persistent

  • 6. EDO indices provide identity

    • Indices are unique

    • Indices span files, federations, DB technologies and geographical locations

    • References to parent EDO’s remain valid when parents are replicated or regenerated

Event Data History David Adams BNL


Requirements cont18 l.jpg
Requirements (cont)

  • 7. EDO’s can be replicated or regenerated

    • Copies have the same data, essential history and index as the original

    • For regeneration, much of the nonessential history will differ

  • 8. Replicated and regenerated EDO’s are equivalent to the originals

    • Either may be provided in place of the original

  • 9. The history for a regenerated EDO should indicate its secondary nature

Event Data History David Adams BNL


Design issues l.jpg
Design issues

  • Object identifiers

    • We require a mechanism to assign a unique index to each EDO and its associated history

    • Here is an example implementation:

      • Use 64 bits (20 years of 109events/yr with 5k EDO’s/event uses 47 bits)

      • A central source serves collections of unused indices to local disks

      • Each job gains exclusive access to a collection

      • Collection hands out unique indices

Event Data History David Adams BNL


Design issues cont l.jpg
Design Issues (cont)

  • 2. Distributing history information

    • Much history info is common to many EDO’s

      • Within a file share data without duplicating it

      • Separate out job and algorithm histories

    • Job history

      • Release version

      • Runtime environment

        • OS, shared libraries and their versions

      • CPU identifier (e.g. hostname)

      • Start time

Event Data History David Adams BNL


Design issues cont21 l.jpg
Design issues (cont)

  • Algorithm history

    • Type

    • Version

    • Name or identifier (Gaudi name)

    • Run time parameters (Gaudi properties)

    • Subalgorithm histories

Event Data History David Adams BNL


Design issues cont22 l.jpg
Design issues (cont)

  • Data history (for each EDO)

    • Event ID

    • EDO

    • Job history

    • Algorithm history

    • Parent EDO’s

    • Global data indices (calibration, alignment, …)

    • Start and stop times

    • CPU time consumed

    • Algorithm return status

    • Data checksum

Event Data History David Adams BNL


Design issues cont23 l.jpg
Design issues (cont)

  • 3. Transient interface

    • For now we define transient classes describing the three types of histories

    • StoreGate converters will make these persistent

  • 4. Historian

    • Provides a convenient mechanism for generating history objects

Event Data History David Adams BNL


Status l.jpg
Status

  • Current implementation

    • All classes in package Control/AthenaHistory

      • JobHistory

      • AlgorithmHistory

      • DataHistory

      • Historian

    • Each class has a component test

    • Builds and tests successfully in 2.4.1

      • Tests must be run by hand

        • (waiting for support from ATLAS/CMT)

Event Data History David Adams BNL


Future l.jpg
Future

  • 1. Modify athena to create history objects

  • 2. Make history objects persist

  • 3. Teach algorithm to specify parents

    • Or should this come from athena?

  • 4. Teach algorithm to return parameters

    • Relevant parameters instead of all properties

  • 5. Design and implement EDO identifiers

Event Data History David Adams BNL


Future cont l.jpg
Future (cont)

  • 6. Add history for secondary objects

    • Flag for replicas

    • More data for regenerated data

  • 7. Missing history

    • Because history never written or was deleted

    • Ancestry chain is broken

    • Merge missing history into that of the child

Event Data History David Adams BNL


Future cont27 l.jpg
Future (cont)

  • 8. Mutable EDO’s

    • Updates

      • If an EDO is updated in a separate algorithm, then history must include the updates

    • Early references

      • Complications if an EDO is updated after it is used as a parent.

        • Child may not be reproducible from the update

      • Reference to parent will need to be extended to include the state of the EDO

      • Or treat each state as a separate EDO

Event Data History David Adams BNL


ad