SA1: Production e-VLBI service

1. SA1: Production e-VLBI service Arpad Szomoru JIVE

2. Outline • e-VLBI science/test runs • Operational improvements • Global e-VLBI: the demonstrations • Hard- and software developments EXPReS Annual Review: SA1

3. e-VLBI science/test runs • 1 Target of Opportunity project (target: nearby supernova) • 1 response to a triggered proposal (target: accreting X-ray binary in a special X-ray state) • 2 exploratory projects (both targets were candidate radio counterparts to so far unidentified gamma-ray source) • 1 Spectral line project (targets: candidate MASER sources) • October deadline only had one triggered proposal • Lack of triggers meant no science runs until March 2009 • ...but, available time was used very well • Different proposal/selection mechanism clearly needed • 3rd year already much more productive EXPReS Annual Review: SA1

4. e-VLBI: operational improvements • Overall improvement in second year: • Huge increase in robustness, reliability and user-friendliness • Not limited to e-operations • Increase of operational data rate from 256 to 512 Mbps • First experiments at ~1 Gbps (with European telescopes) • Demonstrations of global e-VLBI with Chinese and Australian telescopes • Solution of long-haul problems • Inclusion of Effelsberg, Metsahovi, Shanghai, Arecibo (again), TIGO EXPReS Annual Review: SA1

5. e-VLBI: data rate improvements EXPReS Annual Review: SA1

6. Towards global e-VLBI • Long-haul high-bandwidth data transport: • TCP Reno on old linux kernels clearly inadequate • Modern TCP stacks much better, UDP even better, but impossible without modification of Mark5A control code • And can be hostile to other users on open network • Stability of original Mark5 code a serious issue EXPReS Annual Review: SA1

7. Global e-VLBI: hacking time • With several high-profile demos coming up: time to take matters in our own hands • First step: re-enable UDP in Mark5A code • abysmal performance... • Incorporate VLBI-UDP in Mark5 code, in collaboration with colleagues from UMan • Simplified version developed at JIVE, used in both APAN and EXPReS-Oz demo • Complete re-write of Mark5 control software, with rigorous thread control, enabling packet dropping, re-ordering, padding • Used successfully to transfer data at near- line-speed from European telescopes, from Arecibo and, recently, from South Africa and South America • Large development effort, but has paid off in a big way EXPReS Annual Review: SA1

8. The road to China: the demo at Xi’An EXPReS Annual Review: SA1

9. Needed: Reliable transport from standard Mark5 at Shanghai (ancient Linux kernel) Physical connection between two non-communicating Chinese NRENs Some way to bypass the Great Chinese Firewall Interface between correlator and Australian telescope control systems On-the-fly translation from LBA to Mark5B format What could possibly go wrong...... The APAN demo EXPReS Annual Review: SA1

10. Opened up Siberia connection Excellent PR Longest (e) baseline ever APAN results EXPReS Annual Review: SA1

11. The EXPReS-Oz demo • Three Australian telescopes (Mopra, ATCA, Parkes) • 12 hours at 512 Mbps without a hitch • Beautiful detection of SN1987a EXPReS Annual Review: SA1

12. Correlator upgrades • Network upgrade: • All internal JIVE and e-VLBI traffic now through own routers (HP5412zl and HP3500yl-48 ProCurve) • Found quite a few of (serious) bugs in HP equipment • So far, one bug fix out of 5 active cases • Multiple 1-Gbps UDP streams not foreseen by HP engineers? • Dedicated network monitoring platform alongside routers • Removal of dependencies on ASTRON network • Upgrade of Mark5 hardware (motherboards, memory, power supplies) completed • New data acquisition platform (Solaris AMD server, fully interchangeable with redundant control computers EXPReS Annual Review: SA1

13. New modes, feedback and streamlining • Completed: • General speedup of correlator control code • Real-time Mark5 debug info feedback to stations • Re-connecting of individual stations without job re-start • Space-craft tracking mode • Use of Westerbork synthesis data to improve e-VLBI calibration and target selection • Network monitoring tools • Currently under development: • Adaptive observing • On-the-fly fringe fitting • Real-time station-log processing, display of Tsys, status information • Mixed-mode e-VLBI • Integration of Mark5B • 1024 Mbps e-VLBI (and more..) EXPReS Annual Review: SA1

14. New network monitoring tools... EXPReS Annual Review: SA1

15. And new telescopes... EXPReS Annual Review: SA1

16. In summary • Guaranteed high bandwidth through use of an aggressive protocol *and* reliable software • Lightpath technology offers clear advantages; routed connections need negotiations • Convincingly demonstrated that it is possible to overcome long-haul connectivity problems • Simplicity paid off: trivial to connect new EVN stations once they come online EXPReS Annual Review: SA1

17. EXPReS Annual Review: SA1

18. EXPReS Annual Review: SA1

19. SA1: state of deliverables • A total of 15 deliverables were due by month 24, 11 at JIVE and 4 at the University of Manchester • 8 out of 11 JIVE deliverables were completed • Most of the work on the remaining 3 was finished • 3 deliverables that were not due until the last year of EXPReS were completed well ahead of time • The 4 UniMan deliverables were delayed because of the delay in the delivery of the e-Merlin correlator. Most of the IBOB design work has been completed, but tests await the arrival of the first I/O boards, in the summer of 2008 EXPReS Annual Review: SA1

20. Outsourcing • Three workpackages were outsourced to MIT: • SA1.24: Mark5A code modifications • Working version available (though no “official” release yet) • SA1.25: Mark5B code modifications • Dimino (sending) side largely done in-house • Domino (receiving) side developed at MIT, debugging in progress • Extensive engineering support in integration of Mark5B • SA1.20: Improved network applications • mainly done in-house, with MIT support EXPReS Annual Review: SA1

21. SA1: the next 18 months • The focus of SA1 for the next 18 months will be on • Adaptive scheduling • Further integration of Mark5B in EVN operations • Streamlining operational data rate adjustments • Further improving the robustness and flexibility of the correlation process • Mixed-mode operations • Increasing the data rate beyond 1 Gbps EXPReS Annual Review: SA1

22. Specific tasks • Fast/adaptive (re-)scheduling • Transform EVN into a truly flexible instrument • Based on automated download and execution of modified observing schedules - possibly on-the-fly • Reactions from stations vary from “excellent idea” to “over our dead bodies” • But will further improve reliability and responsiveness of EVN • All bits tested and in place, full operational test is planned EXPReS Annual Review: SA1

23. Specific tasks (2) • Data rate adjustments • Packet dropping: • Drop packets at sending side (while preserving headers!) • Pad datastream at receiving side with dummy data • Channel dropping: • Selectively drop frequency channels at receiving side • High demand on cpu, need hyperthreading, right kernels, dual cpu’s • Mixed configurations: • Use different configurations at different stations (limited to 32, 64, 128 etc Mbps) EXPReS Annual Review: SA1

24. Specific tasks (3) • Mixed-mode operations: • Enable all possible combinations of recording (at stations and/or at correlator), real-time and delayed data transfer • Make best use of complete EVN network, including stations that have less-than optimal connectivity EXPReS Annual Review: SA1

25. Specific tasks (4) • Beyond 1 Gbps • 1024 Mbps will simply not fit on 1 Gbps... • Westerbork: use of DWDM equipment, dual 1-Gbps connection, channel bonding • Could be used to Jodrell Bank as well (dual 1-Gbps lightpaths to JIVE) • Onsala, Effelsberg: share 10-Gbps connections with eLOFAR, via VLAN (nearly in place) • Next, Torun? • Most sensitive telescopes in Europe soon at a full 1024 Mbps • And tests ongoing at 4 Gbps, from Onsala EXPReS Annual Review: SA1

26. And soon to come... • Yebes first light (any moment now) • Hartebeesthoek: demo last month at 64 Mbps, S-African universities expect to be connected to Europe at 10 Gbps in 2009 • Arecibo: recent tests at ~512 Mbps • St Croix? • Urumqi: 622 Mbps connection (still in rumor phase) • Japan: talks ongoing about combining VERA telescopes with EVN in e-VLBI pilot project • Australia-China-Japan e-VLBI demo upcoming (at e-VLBI workshop in Shanghai) EXPReS Annual Review: SA1

28. SA1- Past 12 month overview SA1 Deliverables EXPReS Annual Review: SA1