1 / 21

dfos: tutorial

dfos: tutorial. www.eso.org/~qc/dfos/. Lesson 1: dfos - vision and architecture Directory structure Installation Tips and tricks. Lesson 2: Association. Lesson 3: Operations. Vision. Complexity ... ... comes by itself: Simplification ... ... needs efforts

graham
Download Presentation

dfos: tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. dfos: tutorial www.eso.org/~qc/dfos/ • Lesson 1: • dfos - vision and architecture • Directory structure • Installation • Tips and tricks • Lesson 2: • Association • Lesson 3: • Operations

  2. Vision • Complexity ... • ... comes by itself: • Simplification ... • ... needs efforts • The answer: dfos = Data Flow Operations System • www.eso.org/~qc/dfos/

  3. dfos tools dfos regime user regime config files Vision • To obtain … • common understanding of tools and workflows • contributions from everyone possible • easier multi-tasking • cross-instruments training possible • Principle: • all workflow steps in common dfos tools • all instr-specific info in config files

  4. Architecture Tools for two workflows: • daily workflow • packing workflow Level 1 tools: called by user Level 2 tools: called by level 1 tools Interaction: • command line • graphical user interfaces: DFO monitor; SM monitor (self-managing HTML pages)

  5. Architecture Inter-tool “communication”: status flags • DFO_status; SM_status etc. (read by monitors) • possible to be read by wrapper tools $DFO_MON_DIR/DFO_STATUS:

  6. Architecture central element of dfos: the Association Block AB (--> Lesson 2)

  7. Architecture daily workflow sketch:

  8. Directory structure • dfos needs well-defined directory structure • environment variables • many have DATE substructure, with DATE as 2005-01-14: • e.g. $DFO_RAW_DIR • others are flat • e.g. $DFO_MON_DIR • one is more complex: • $DFO_SRV_DIR/<run_id>/<DATE>/raw etc. • $DFO_SRV_DIR/<run_id>/<obs_id>/sci_raw etc.

  9. Installation Get a complete installation of dfos (daily workflow) in four steps: 1. Getting started 2. Initial tool set 3. Creating ABs 4. Finish installation this is where you can get without ABs

  10. Step 1: Getting started Obtain a dfos installation: • prepare the initialization file .dfosrc • prepare the dfos directory structure • download dfos tools • prepare the configuration files 1. Prepare your account and install the initialization file .dfosrc  download .dfosrc into $HOME  edit as appropriate  create directory structure as defined  make sure .dfosrc is sourced properly 4x : ok, continue

  11. Step 1: Getting started 2. Install dfosExplorer (by hand):  download and untar  execute 2x : ok, continue 3. tune your browser to become more interactive (tune your browser) 4. Read the documentation (‘help’ and ‘dfos’)

  12. dfosExplorer use action button to download edit config file

  13. Step 2: Initial tool set 5. Install dfosInstall (using dfosExplorer) 6. Install: • dfosLog • dvdMonitor and dfosCron • dfosInstall: new installation/upgrade • new installation: installs everything, incl. template config file (old one saved!) • upgrade: config file is installed in $DFO_CONFIG_DIR/template (no overwrite!) • config files: always come as template; some need no editing at all, others need heavy editing. Always read documentation, all configurable parameters are documented! Remember: config files are your contribution and your responsibility.

  14. Step 2: Initial tool set 7. Install: dataclient, fitsreport, createReport, checkDownloads • these tools define the pre-processing part of dfos • they don’t need ABs • dataclient: the tool for header/fits file downloads implicit downloads (specify type, date, mode): • dataclient -t h -d 2005-01-31: database headers • dataclient -t r -m all -d 2005-01-31: all raw fits files explicit downloads (specify ARCFILE names): • dataclient -t r -m l -l my_list.txt -o /data03/data/my_outpath/ • fitsreport: the workhorse for fits key reading • used by various different tools • flexible (formatted/unformatted), performant • createReport: general reporting tool (dfo reports, user reports)

  15. Step 3: Creating ABs We are here: • Next: AB creation, processing workflow • Today: no details about association, just assume you can do it ... • ABs: text file with all information about a set of raw frames necessary to: • process and/or • pack • --> Even without a pipeline, you want to work with ABs.

  16. Step 3: Creating ABs 8. Install: createAB (comes with: helpAB, listMcalib) • it uses fitsreport to create lists: raw_list, vcalib_list, mcalib_list (all under $DFO_MON_DIR) 9., 10. create the config files (Lesson 2) • prepare: • know your raw data types (DPR keys) and product types (PRO.CATG) • know the match keys (‘setups’) • know your pipeline (recipe names) • know your DRS • sketch your calibration map (piece of paper is fine)

  17. Step 4: Finish installation 11. Install: createJob, processAB; getStatusAB • supported DRS: esorex, RBS, REI 12. Install: measureQuality, certifyProducts • prepare your QC procedures (MIDAS prog is attached as example) • QC procedures are as instr-specific as pipelines, no attempt made to include them into dfos • measureQ is a wrapper to call these procedures • certifyP is also a wrapper around a user-provided procedure to call the QC1 plots (example provided) • if you have no procedures yet: skip that part

  18. Step 4: Finish installation 13. Install: moveProducts, renameProducts, updateAB • moveProducts calls renameProducts, updateAB, listMcalib • listMcalib updates the mcalib list, using fitsreport • cdbIngest is not yet called by moveProducts, TBD 14. Install: updateDP, finishNight • updateDP reads ABs for packing • supports: DATE and OB scheme • has a graphical monitor • also needed if pipeline support is lacking • finishNight: cleans up (almost) everything

  19. Step 4: Finish installation updateDP:

  20. Tips and tricks Tune your browser • follow the description to make your browser an interactive GUI: • launch shell scripts by clicking on a button • open an editor to edit your config files with the browser • used by the dfos monitor tools: dvdMonitor, dfosExplorer, dfoMonitor, updateDP, smMonitor, relMonitor • after tuning, mozilla will execute any script with extension .ash or .esh just by clicking

  21. Tips and tricks Cronjobs • delegate some work to cronjobs: dvdMonitor, header download, createReport, dfosExplorer • use dfosCron to call these tools • test the cronjob and the way it sources .dfosrc! FAQ • read that web page before asking FAQs ...

More Related