slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The ROB SDO data system PowerPoint Presentation
Download Presentation
The ROB SDO data system

Loading in 2 Seconds...

play fullscreen
1 / 23

The ROB SDO data system - PowerPoint PPT Presentation


  • 105 Views
  • Uploaded on

All of the Sun all of the time: Distributing 1TB/day from the Solar Dynamics Observatory satellite, 24/7 for 5+ years. The ROB SDO data system. David Boyes, Véronique Delouille, Benjamin Mampaey, Tobias Berghoff, Cis Verbeeck, Jean-François Hochedez (Royal Observatory of Belgium).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The ROB SDO data system' - heba


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1
All of the Sun all of the time: Distributing 1TB/day from the Solar Dynamics Observatory satellite, 24/7 for 5+ yearsThe ROB SDO data system

ROB for BNC 2009 Brussels

the presentation
ROB for BNC 2009 Brussels

David Boyes, Véronique Delouille, Benjamin Mampaey, Tobias Berghoff, Cis Verbeeck, Jean-François Hochedez (Royal Observatory of Belgium)

The Presentation
  • The science - studying the Sun
  • The system - satellite, network, data centres
  • In practice - getting it all to work

The people

why look at the sun
Why look at the Sun?
  • Weather - it affects us but can be forecast
  • Science - there is still a lot to find out!
    • why is the solar corona so hot
    • what drives mass ejections
    • why is there an 11 year cycle
    • … and much more

ROB for BNC 2009 Brussels

why have satellites
Why have satellites?
  • The earths atmosphere blocks a lot - UV and above
  • At these wavelengths the structure of the sun is revealed

ROB for BNC 2009 Brussels

what are the effects on earth
What are the effects on earth?
  • Radiation - effects are immediate, goal is to predict them
  • Particles - arrive in hours or days - we can give warning
  • This is solar weather forecasting - ROB is a Regional Warning Center

Graphic : NASA

ROB for BNC 2009 Brussels

what s up there
What's up there?
  • Telescopes to observe the Sun's atmosphere at multiple UV wavelengths (AIA)
  • Telescopes to measure specific wavelengths to allow calculation of magnetic fields and seismic activity (HMI)
  • Wide band UV sensor to measure total UV spectrum (EVE)

ROB for BNC 2009 Brussels

some numbers about sdo
ROB for BNC 2009 BrusselsSome numbers about SDO
  • A massive increase in data quantity and precision - 1000 to 10000 times as much data as current satellites
  • Flies at 38 000 km, geosynchronous orbit
  • AIA - images at 10 wavelengths from visible to 131Å, one image every 1.25s
  • HMI – one magnetic image every 45s
  • EVE – irradiance time series from 10 to 1050Å
  • Images are 4Kx4K - 32MB per image
  • A lot of data - more than 1TB/day
the challenges of sdo
ROB for BNC 2009 BrusselsThe challenges of SDO
  • Huge bandwidth
  • Lots of data to be made available
  • Too much data for humans to absorb
the solution
ROB for BNC 2009 BrusselsThe solution
  • A worldwide network of data stores holding current quarter and popular data
  • Joined by high-speed network
  • Pushing a full copy of data to as wide an area as possible in compact form
  • Software system (netDRMS with internal PostgreSQL) at each data store provides virtual storage for file requests from users
  • Transparent access to any data, if needed going down to original source data
  • Local users have the impression they have file access
  • Web based mediation for remote use interface
  • Automatic processing by high performance computing
what s down here
ROB for BNC 2009 BrusselsWhat's down here
  • Ground station - main station at White Sands
  • Mission Operations Center at Goddard
  • Joint Science Operations Centers (JSOC) - Stanford and Colorado
  • Knowledge base – Lockheed, Virtual Observatory - Harvard
  • Storage at White Sands, JSOC and Data Centres
  • Compute clusters and data servers at Data Centres
  • A network of Data Centres...
the data centres
The Data Centres

ROB for BNC 2009 Brussels

what this enables
ROB for BNC 2009 BrusselsWhat this enables
  • Many groups working in parallel on the unprecedented flow of data
  • Simultaneous access and processing of bulk data in many high-performance systems
  • Online access for forecasters to complete data to refine their techniques
  • Completely open and low cost access to all data for both researchers with specific interests and for researchers with limited budgets
does it work
Does it work?
  • Yes it does – e.g. two weeks 320Mb/s from Harvard

ROB for BNC 2009 Brussels

network requirements
ROB for BNC 2009 BrusselsNetwork requirements
  • Throughput
    • One set of data takes around 200Mb/s
    • Requires 320Mb/s to handle catch ups
    • Practical limit is network chain topology
  • Availability
    • More than five year, probably ten year operation
    • 24/7, 365
    • Must maintain full performance for backbone data system even with subsystem failures
in practice bandwidth delay product
ROB for BNC 2009 BrusselsIn practice - Bandwidth-Delay product
  • There are simply a lot of Bytes in the cable - this is the Bandwidth-Delay (BD) product
  • Problem with the TCP protocol is that buffer size >= 2 * bandwidth * delay and the actual size is adaptive
  • For example 200Mb/s and 0.1s → 5MB, and you need about twice that for adaptation.
    • Standard Linux buffer size is 64K!
  • Plus you can run into congestion control limits – designed to share traffic fairly!
in practice bd product
ROB for BNC 2009 BrusselsIn practice - BD product
  • Fixes …
    • Use an improved scp (HPN-scp)
    • Use multiple sessions
    • Use another protocol
    • Use a tool which combines these (e.g. GridFTP)
  • We use multiple sessions in user space
    • Raw bandwidth tests used many more than needed
    • Production system has tool which interfaces with the data system
in practice routing
ROB for BNC 2009 BrusselsIn practice - routing
  • Check it - you might be surprised
    • Different networks have quite legitimate different behaviour
  • What didn't get noticed with e-mail and web pages can still be a problem
    • The odd few minutes for e-mail don't get noticed
    • Low speed at 2am probably won't get noticed
in practice reliability
ROB for BNC 2009 BrusselsIn practice - reliability
  • Can't use terms like guarantee – things will go wrong
  • Can't be qualitative – this is way beyond normal hardware reliability
    • You still need quality, duplication, spares and conservative ratings
  • Must get quantitative – failure analysis and point of failure identification
    • Time to repair (night shifts!) is critical
  • Must be able to detect failures
    • A single failure will not show up in system performance
in practice reliability1
In practice - reliability
  • This is how the Belnet connections at ROB deliver reliability
  • Single failures do not affect data flow, regardless of which HA node is active
  • You must check that no failure has occurred

ROB for BNC 2009 Brussels

in practice last mile
ROB for BNC 2009 BrusselsIn practice - last mile
  • It's here you will have the most problems
    • Both ends will need work
    • Firewalls
    • Routers
    • Just where is the cable really
    • A server is not quite as good as the manufacturer said
  • Again, situations which might have gone unnoticed will make themselves known
  • But you are right there to fix them...
slide21
The data network is running

The data transfer system is testing

The system is being documented

So ... it's looking good

Where it's at

ROB for BNC 2009 Brussels

further reading
ROB for BNC 2009 BrusselsFurther reading
  • SDO at the ROBhttp://wissdom.oma.be
  • Belnethttp://www.belnet.be
  • SDOhttp://sdo.gsfc.nasa.gov
  • ESnet Network Performance Knowledge Base http://fasterdata.es.net
  • High Performance Enabled SSH/SCP http://www.psc.edu/networking/projects/hpn-ssh
thanks for your interest and success with your projects
Thanks for your interest

and success with your projects

The ROB SDO data system

ROB for BNC 2009 Brussels