1 / 30

VLBI Storage Manager (VSM)

VLBI Storage Manager (VSM). Matteo Stagni – Mauro Nanni. Network and Storage in the future of the VLBI. Attended antenna with local storage system (B) Groups of antennas with a central storage system and a local correlator (C) Unattended antenna with local storage system (D)

mulan
Download Presentation

VLBI Storage Manager (VSM)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VLBI Storage Manager (VSM) Matteo Stagni – Mauro Nanni

  2. Network and Storage in the future of the VLBI • Attended antenna with local storage system (B) • Groups of antennas with a central storage system and a local correlator (C) • Unattended antenna with local storage system (D) • Antennas that use a remote storage (E or A ) Cloud ? • Antennas with poor network connection that use disk-pack

  3. Disk Pack - vs - Storage Systems Disk Pack: • Field System manage the Disk Packs substitution • Disk packs are empty and every Pack has a fixed capacity Storage Systems: • Different hardware characteristics • Different capacity (40-120 TBy) that change in time: • Presence of previous observation data • Space reserved for different observation programs • In a session for one antenna can be requested more storage systems • More antennas can store data on one storage system (low speed !) • Storage systems near the antennas or connected by high speed net

  4. VSM Storage Units Unit Box Capacity: 40-100 TBy Network: 10 Gbit/sec File system: standard Writing speed : more than 4 Gbit/s Space on the unit: Used: data of previous observations Reserved : allocated for future obs. Free: can be allocated

  5. Storage Units: Our Choice • Supermicro Systems: • Motherboard: X8DTH-IF • CPU: 2 Intel XEON E5620@2.4GHz • RAM: 12 GB • RAID: 3WARE 9750DR • Disks: 24 + Os • Network: 10 Gbit • OS: Scientific Linux 6.2 Writing Raid 5 XFS Filesystem: Command dd : 1.2 GB/s Vlbistreamer : 6135 Mbit/s (raid 5 degraded: -10% in speed)

  6. Write test on different filesystems Tests were run on a 12 disk RAID5 array with 22 TB space

  7. Vlbi Storage Manager

  8. The Vlbi Storage Manager (VSM) The new storagedevicerequire a system for management of the data on disk: • management of disk space • reservingresources for the future sessions • overseeing the capture and registration • data transfer to the correlator Our choices: • Hardware : commercial “Off The Shelf” components • Operating System: Linux • Standard file systems • Database Mysql • Shell Bash and PHP Different organizations can use different architectures and adopt the solution at a different pace

  9. System Architecture

  10. VSM general architecture VSM Network-DB: Session data: c/o the correlators or central organizations VSM Station-DB: Storage station data: c/o Radio observatory Storage stations get data of the sessions from the network DB

  11. 0 - Session sckedules are available 1 - Sckedules data are written in VSM-Network DB 2 – Station Manager get data in the VSM-StationDB 3 – Station Manager verify space (vsm_analize) 4 – Station Manager reserve the space (vsm_reserv) 5 – Station Manager create rec-sked (vsm_sched) 6 – Rec-Sked starts vlbi-stream at observation time

  12. Network environment

  13. Vlbi antennas schedule This table is modeled after the present text file worked out in Bonn to coordinate european VLBI observations

  14. Each observation is registered In the antenna table

  15. Each network station management host can be reached and its status monitored…

  16. Network admin can conveniently update the vlbi observations through a FORM

  17. System Architecture

  18. Station environment

  19. VSM Storage Station The “VSMstorage stations" are the structures that manage one or more "storage unit" under the supervision of the technical staff of the organization A storage station can manage one or more well connected antennas • A VSM storage station is made by: • One or more storage units • The VSM Software • The VSM Database of space usage in the storage units • The VSM Database of the local experiments

  20. The VSM System core concept is to manipulate the antennas’ scheduled and present data record on different MySQL databases. Dual implementation for the local admin management: - CLI (Command Line Interface) Bash - WEB-BASED Interface Php

  21. CLI Interaction Command set:

  22. WEB-BASED Interaction

  23. WEB-BASED Interaction A php replica of the bash command set… • In detail: • Storage status • Reservespace • Schedule observations

  24. WEB-BASED Interaction Through Storage status we get the space occupied on the filesystem by experiments on machines

  25. WEB-BASED Interaction Reserve space has done its job and checked if there is enough room for the upcoming observations and now we’re ready to schedule them…

  26. WEB-BASED Interaction The jobs are ready to be scheduled on the machines with the appropriate command at the selected time

  27. WEB-BASED Interaction The jobs are now waiting on the machine in the at queue

  28. A primary PoP of the NREN at the IRA HQ building With Lambda DWDM device

  29. Network for VLBI in Italy

  30. Radioastronomy network TN1 TN1 UD1 UD1 MI4 MI4 CO CO TN TN MI2 MI2 MI1 MI1 UD UD PD PD TS TS TO TO PV PV MI5 MI5 MI3 MI3 FE FE FE1 FE1 MI1 MI1 MI1 MI1 BO1 BO1 BO2 BO2 GE GE FI FI AN AN PI PI 2x10G 2x10G AN1 AN1 PG PG IRA IRA - - INAF BO INAF BO AQ1 AQ1 BO1 BO1 RM2 RM2 PE PE 3x10G 3x10G Civitavec. Civitavec. AQ AQ Termoli Termoli 10G 10G G.652 G.652 FG1 FG1 RM1 RM1 CB CB 10G 10G 10G 10G BA BA SS1 SS1 Olbia Olbia BR BR NA NA LE LE SA SA MT MT PZ PZ CS CS Oss. Oss. Oss. Oss. Oss. Oss. CA CA CZ CZ CA1 CA1 Noto Noto (SR) (SR) S.Basilio S.Basilio (CA) (CA) Medicina Medicina (BO) (BO) Lamezia Lamezia G.654 G.654 PA PA ME1 ME1 RC RC Mazara Mazara CT CT

More Related