1 / 16

Staging Archived Data at DESY

Staging Archived Data at DESY. Albert.Kagarmanov@desy.de December 2008. Channel Archiver. Currently we permanently archive ~ 4000 PVs We are using ChannelArchiver for initial archive They devided to 10 sub-archive correspondent to our main facilities – MKS, MKK, CTA, CMTB, etc

zeus-alston
Download Presentation

Staging Archived Data at DESY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Staging Archived Data at DESY Albert.Kagarmanov@desy.de December 2008

  2. Channel Archiver • Currently we permanently archive ~ 4000 PVs • We are using ChannelArchiver for initial archive • They devided to 10 sub-archive correspondent to our main facilities – MKS, MKK, CTA, CMTB, etc • ChannelArchiver working in our SUN-FIRE Solaris FileServer • ChannelArchiver produced 2 Gb raw binary data pro week

  3. Archive Daemon web interface

  4. Channel Archive Limitations • “Archive is perfect but how to extract data from it?”  • raw-data saved in model “one directory/day/sub-archive” • i.e. /CAR/mkk/2008/11_24 • Channel Archive working fast and relaible for store data, but extracting from binary file is not so fast. Channel Archive produced one unique master master_index file for real-time indexing data using balanced R-Tree theory • Really slow if master_index keeps all tracks for more than 3 month history

  5. Our Long term Archiving Schema • Data from ‘now‘ till 2 month before is collected under normal ChannelArchiving Schema   • BinaryData 2 month old  till 12 month old  is moved under another directory CAR_Save   • In parallel, data which 1 month old convert to SDDS-format. • SDDS- well-known „Self Describing Data Set“-format is APS-Argone special format for physical data store and handeling : www.aps.anl.gov/asd/oag/software.shtml • SDDS-archive is our “final” archive format. All data from 1996-2008 permanently avaiable under this format

  6. ChannelArchiver2SDDS converting details: • This is periodical (1 time pro week) script • Pipe of 2 tasks: • Standart ArchiveExport command which provide ASCII output from ChannelArchiver • ASCII2SDDS DESY C++ program which from ascii input create SDDS file • Source for ASCII2SDDS : • www-mks2.desy.de/content/e4/e40/e41/e12077/e12085/e21187 • Few years ago we also reduce old data using special DESY developed algorithms, now we’re not use it since we’re don’t problem with disk space: Source for DataReduction avaiable under www-mks2.desy.de/content/e4/e40/e41/e12077/e12085/e21382

  7. ChannelArchiver2SDDS converting details: IOC Now CA Disk current Archive mv longTerm Archive Disk ASCII2SDDS SDDS Disk compress Reduction/ZIP Disk Time

  8. SDDS-format and AAPI client/server • SDDS-data store in model • I file pro PV pro month: • i.e. /data/sdds/2008/08/wienerVME:memUsed_ai.gzip • SDDS-lib support auto-unzip (ZIP-ratio ~ 50%) • Extracting SDDS-data tool is AAPI-server • AAPI-server is C/C++ TCP/IP daemon program which waits request for archive-data from AAPI-clients • AAPI-server can handle: • ChannelArchiver Binary Data (last few months) • SDDS-data (1-12 year old archive data which converted to SDDS-format) • DESY-specific TINE and DOOCS archive data (another DESY Control Systems • Main AAPI-clients is CSS-DataBrowser and old StripTool

  9. SDDS-format and AAPI client/server • Source for AAPI-server available under www-mks2.desy.de/content/e4/e40/e41/e12077/e12082 • Curent distribution is AAPI-2.7.tar.gz • Java-client lib can be found in CSS CVS distribution

  10. Gap between live and ChannelArchiver data • ChannelArchiver collect data in memory and flushed data to a disk periodicaly (DESY-default=5 min) => Initial ChannelAtchiver browsers have small gap (~5 min) between live- and archive-data.

  11. Small gap in data How to avoid this gap: Our idea is to store last ~200 values in IOC, in some sort of waveform record ( IOC-cache) -- archiveRecord.

  12. IOC-archive cache -- ArchiveRecord • Here, at DESY archiveRecord was developed. • For each normal PV we have PV_h-’waveform-like’ record where we store last N (default=200) PV value,status and timestamp from normal PV. • CSS-dataBrowser and old StripTool can handle archiveRecord value • Gzip distribution is http://www-kryo.desy.de/ documents/EPICS/DESY/General/Archiver/DesyStripTool/archRec.tar.gz

  13. IOC-archive cache -- ArchiveRecord

  14. CSS Databrowser • Most robust EPICS archive data browser • Support different archive data extension points – independent archive data types: • ChannelArchiver raw binary data • AAPI protocol (ChannelArchiver + SDDS) • archiveRecord data • Direct DESY TINE archive data • …

  15. CSS Databrowser • xnds://www-kryo.desy.de/applets/LACA/cgi/ArchiveDataServer.cgi - is standard ChannelArchiver ArchiveDataType • aapi://epicsh.desy.de:4053 - is DESY AAPI-server ArchiveDataType • archiveRecord:// - is DESY AAPI-server ArchiveDataType …

  16. CSS Databrowser • Preference Page of DataType extension point configuration The end

More Related