bytestream components n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
ByteStream Components PowerPoint Presentation
Download Presentation
ByteStream Components

Loading in 2 Seconds...

play fullscreen
1 / 10

ByteStream Components - PowerPoint PPT Presentation


  • 112 Views
  • Uploaded on

ByteStream Components. Jack Cranshaw February 20, 2012. ATLAS Code Base. TDAQ releases Contain the basic structure of blocks in the ByteStream format. Sets the content. Currently on tdaq-common-01-09-00

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

ByteStream Components


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
bytestream components

ByteStreamComponents

Jack Cranshaw

February 20, 2012

atlas code base
ATLAS Code Base
  • TDAQ releases
    • Contain the basic structure of blocks in the ByteStream format. Sets the content.
    • Currently on tdaq-common-01-09-00
    • Releases occur roughly every 6 months and must be approved by trigger group and online.
    • Attached to the ATLAS Offline code in external package DataCollection.

I/O and Persistence Evolution Workshop

atlas code base1
ATLAS Code Base
  • Event/ByteStreamData
    • Classes for data storage (ROB, RawEvent, metadata)
  • Event/ByteStreamCnvSvcBase
    • Utility and base classes used by detector Cnv, address providers and data providers
    • Converters in detector specific packages like for normal T/P.
  • Event/ByteStreamCnvSvc
    • Input and output services.
    • Utility programs such as AtlCopyBSEvent, AtlListBSEvent.
  • Event/ByteStreamStoragePlugins
    • Plugins to map different storage technologies onto the same basic converters: castor, xrootd, dcache, gfal.

I/O and Persistence Evolution Workshop

dependencies
Dependencies

Control/DataModel

Event/ByteSreamData

External/DataCollection

Database/AthenaPOOL

External/AtlasROOT

Event/ByteStreamCnvSvcBase

Event/ByteStreamStoragePlugins

Event/ByteStreamCnvSvc

I/O and Persistence Evolution Workshop

bytestream input in athena
ByteStream Input in Athena
  • Handled by changing EventSelector used by serviceMgr.
  • Mostly handled by job options in ByteStreamCnvSvc/ReadByteStream.py. User just sets file name to EventSelector and ByteStreamInputSvc.
  • ReadByteStream.py
    • Adds ByteStreamEventStorageInputSvc, ROBDataProviderSvc, EventSelectorByteStream, ByteStreamCnvSvc, ByteStreamAddressProviderSvc, and ProxyProviderSvc to svcMgr.
    • Sets EventSelector property ByteStreamInputSvc.
    • Sets EventPersistencySvc property CnvServices.
  • TAG input setup different and handled directly in RecExCommon_topOptions.

I/O and Persistence Evolution Workshop

bytestream output in athena
ByteStream Output in Athena
  • Somewhat more sophisticated python contained in ByteStreamCnvSvc/WriteByteStream.py.
  • WriteByteStream.py
    • getStream method returns an AthenaOutputStream.
    • Two types of output: EVENTSTORAGE, TRANSIENT.
    • EVENTSTORAGE (write out)
      • Adds ByteStreamEventStorageOutputSvc to svcMgr.
      • Sets some properties by parsing output name.
    • TRANSIENT (multi-stage)
      • Adds ROBDataProviderSvc and RDP_OutputSvc to svcMgr.
      • Sets EventPersistencySvc property CnvServices since it is faking input.

I/O and Persistence Evolution Workshop

comparison to pool apr root
Comparison to POOL/APR/ROOT
  • BS Input
    • ByteStreamCnvSvc. ReadByteStream
  • BS Output
    • ByteStreamCnvSvc.WriteByteStream
    • Use C++ class AthenaOutputStream from AthenaServices
    • Only handling event data.
    • Itemlist handled in RecExCommon_topOptions
  • POOL Input
    • AthenaPoolCnvSvc. ReadAthenaPool
  • POOL Output
    • OutputStreamAthenaPool. MutipleStreamManager
    • Use python class AugmentedStreamBase
    • Integrated stream handling event data and metadata.
    • Itemlist handled in files in RecExPers package.

Differences on output complicate maintenance

I/O and Persistence Evolution Workshop

job options and packages
Job Options and Packages
  • As noted in the POOL-ByteStream comparison there are differences, primarily on output.
  • There are also a variety of hacks (for most output streams) in RecExCommon_topOptions which could benefit from cleanup.
  • Some pieces of code which seem general are placed in packages which are meant to be technology specific.

I/O and Persistence Evolution Workshop

testing
Testing
  • ByteStreamCnvSvc
    • Tests of utilities AtlCopyBSEvent, AtlListBSEvent
  • AthenaPoolExampleAlgorithms
    • Tests of input and output using ReadByteStream and WriteByteStream.
    • Also tests I/O with TAG as input.
  • RecExAnaTest
    • Tests of usage in RecExCommon using Reco_trf.

I/O and Persistence Evolution Workshop

summary
Summary
  • Concerns
    • There are a variety of hacks which need to be cleaned up if possible.
    • Some of the processes, such as parsing filenames seem brittle.
    • The differences in output methodologies used in RecExCommon complicate maintenance.
    • The package structure (or placement of code) is not logically consistent everywhere.
    • The treatment of metadata is becoming more complex.
  • The system is currently working other than a few metadata issues.

I/O and Persistence Evolution Workshop