Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department

1. Dominik Stokłosa Poznań Supercomputing and Networking Center, Supercomputing Department INGRID 2008Lacco Ameno, Island of Ischia , ITALY, April 9-11 Workflow management in Remote Instrumentation Infrastructure – based on e-VLBI experiences

2. Introduction to the e-VLBI PSNC in EXPReS - FABRIC Grid – VLBI design & technology Summary Outline

3. Introduction to the e-VLBI • VLBI is a technique, in which physically independent and widely separated radio telescopes observe the same region of sky simultaneously, in order to generate very high-resolution continuum and spectral-line images of cosmic radio sources • Telescopes are usually separated by thousands of kilometres • Data from each telescope are digitally sampled and stored locally, using high-capacity magnetic tape systems and magnetic disk-array systems • Data are sent and correlated at the central point (JIVE) • The total flow of data into the central processor is approximately 10-100 Terabytes per single observation, after processing this is reduced to 10-100 Gbytes.

4. Introduction to EXPReS • EXPReS – the objective is to create a production-level “electronic” VLBI (e-VLBI) service, in which the radio telescopes are reliably connected to the central data processor at JIVE via a high-speed optical-fibre communication network. • Project Details • Three years, started March 2006 • International collaboration • Funded at 3.9 million EUR • FP6, Contract #026642

5. PSNC in EXPReS • EXPReS – a Real-time e-VLBI Radio Telescope - JRA1: Future Arrays of Broadband Radio-Telescopes on Internet Computing (FABRIC) • - Grid – VLBI collaboration • - Grid Workflow management • - Grid Routing Creating solution for incorporating Grid resources for distributed correlation using existing infrastructure.

6. Once upon a time (1) • Everything was slow • Telescopes collected data on tapes • Sent via postal mail • Hard drive arrays slightly improved the situation • The entire cycle could easily require 6months or more

7. Once upon a time (2) • Hardware correlator; the EVN MkIV data correlator at JIVE • dedicated, purpose designed/built hardware • a super computer; ~100 T ops/sec

8. Today / In the near future • Data can be transferred over the network • Each stage of the process can be speeded up • GRID resources • Software correlator e-VLBI - electronic VLBI

9. e-VLBI System - architecture

10. e-VLBI System – control flow

11. Workflow Manager module (WFM)

12. CCF – correlator configuration file

13. WFM – phase 1 Definition of radio telescopes – automatically based on theobservation schedule

14. WFM – phase 2 Definition of file servers (each file sever is responsible for capturing data from RT)

15. WFM – phase 3 Definition of correlation nodes and data flows between components

16. WFM – properties Definition of resource properties

17. WFM (1) • The WFM application has several purposes: • Workflow creation – a user is given a list of building blocks. Each block represents the available resource. It is the user’s responsibility to design and construct the experiment’s workflow from the available building blocks. • Workflow submission – the scenario prepared in the previous step can be submitted to the system. • Workflow management and monitoring – allows to manage the executed workflow.

18. WFM (2) Obrazek z WFM -> przeplywy danych

19. Link definition

20. Type definition Radio telescope File server Correlator

21. Properties definition • Dynamic forms • Parameters stored in Json format

22. PSNC in EXPReS EXPReS – a Real-time e-VLBI Radio Telescope - JRA1: Future Arrays of Broadband Radio-Telescopes on Internet Computing (FABRIC) - Grid – VLBI collaboration - Grid Workflow management - Grid Routing Creating solution for incorporating Grid resources for distributed correlation using existing infrastructure.

23. Thank you for your attention http://www.expres-eu.org/ d.stoklosa@man.poznan.pl EXPReS is made possible through the support of the EC, 6th FP, Contract #026642