Utilising Large Astronomical Datasets

1. Utilising Large Astronomical Datasets Brian Schmidt The Research School of Astronomy and Astrophysics Mount Stromlo & Siding Spring Observatories

2. The Golden Age of Astronomy • The last decade has seen Astronomers: • Measure the age of the Universe • Discover the first extra-solar planet • Map the Cosmic Neighbourhood • Figure out how much and what makes up the Universe • Get a good idea on the ultimate fate, and beginning of the Universe

3. What is Left to do? • Observing and understanding the first generation of Stars and Galaxies • Directly detecting and studying Extrasolar Planets • What is the Dark Energy? • What is the Dark Matter?

5. > 1 Billion Dollars

6. Astronomy Cannot Afford to Live on these Instruments Alone • Technology provides the ability to create large datasets • Sift through rare events, or objects, or add the information • of a billion objects together, rather than observing a few objects very • well. • To Succeed, you must have • Well chosen Scientific Goals • Carefully chosen Experimental Plan • Need to work with and beyond Cutting Edge technology – both hardware and software.

7. Worked Example- A large Optical Survey of The Southern Sky Observing and understanding the first generation of Stars and Galaxies – Find the Brightest First Galaxies (Quasars) in the sky, and study these. Challenge: only about 10 useful objects in the sky for the current 8m class telescopes.

8. Worked Example- A large Optical Survey of The Southern Sky Directly detecting and studying Extrasolar Planets- Challenge…Need to monitor millions of stars on the timescale of hours to see the eclipse… Venus 2004 Venus 1882

9. Worked Example- A large Optical Survey of The Southern Sky What is the Dark Energy? – As photons from the Cosmic Microwave Background travel through the gravity wells of galaxies, they compress on the way in, and stretch, on the way out – this process is asymmetric in one of the two favourite forms of Dark Energy. Challenge, need more than a billion galaxies with which to compare to get enough statistics…

10. Worked Example- A large Optical Survey of The Southern Sky What is Dark Matter? – Find the most distant set of stars in our own galaxy, and use them as tracers of the gravity. How far does the Dark Matter Extend, is it spherical? Challenge, need to sift out the roughly 5000 stars that are bright enough to be studied in the outer reaches of our Galaxy.

11. Great Melbourne Telescope: • Equipped with state-of-the art CCDs to • Provide 32 Million Pixels of data every 90 seconds • Over 5 years, a 5 colour 24 Terabyte Image of the entire southern sky at 3 epochs

13. Great Melbourne Telescope automated 2000 • Installing Weather Monitoring system ($15000 + 2 people months) • Installing computer controllable switches on all systems ($15000 + 2 people months) • Moving from Liquid Nitrogen to Closed Cycle Cryogenic System for Instrument cooling ($30000 + 4 people months) • Control Software to monitor weather and control telescope (1 person year) • Software Scheduler to decide what to observe (2 person months) • Quality Monitoring software (2 person months) • New Imaging system ($350000 +2 people years)

15. A new Southern Survey Designed from Scratch…So Huge Field of View…7 square degrees (50 times full moon) 300 Million Pixels of information every 70 seconds Entire sky in 10 nights Over 2 years, A map of the entire sky in 6 colours, with information at approximately 10 epochs. Available to the entire community.

16. The New Telescope

17. Survey Size: Over 2 years 140,000 300,000,000 pixel images = 115 Terabytes of information…

18. Factor of 10 compression, noise increased by 5%

19. 140,000 300,000,000 pixel images = 115 Terabytes of information… but can store this onto < 15 Terabytes ($40,000 for a disk array of this size) Or store on a dedicated Mass storage system (e.g. ANU/APAC Supercomputing facility’s Terabyte storage array)

20. Robust Pipelines • Where in the sky is this? • Where are the objects? • How bright are the objects? • What shape are the objects? • What is bad information? • Beg, Borrow, and Steal from The past 20 years of Astronomical work. No black boxes, nothing proprietary!

21. Data Analysis Flow: Current Pipeline takes approximately 1 hour to complete on a 2.5GHz processor on a single 300,000,000 pixel image. Fortunately we have nighttime and bad weather! On A long winter’s night 500 images would be taken, So total computational power required for analysis is 22 2.5GHz processors – (roughly another $40,000) In 3 years time – only 8 of the newly released 8GHZ Intel Tritium (they are hot!) processors will be required.

22. Potential Problems: • Funding – small potatoes compared to big telescopes, but still big bickkies for grants available to Australian Researchers • Automated systems and instruments are non-trivial to implement. It takes a lot of time of effort to make them trouble free. • Software – although computers can handle the 113 Terabytes, humans cannot. Must have robust pipelines that work completely without human intervention.

23. There are scientific opportunities to those who are willing to tackle large data sets. But requires carefully planned scientific outcomes like any other experiment. Usually Requires Researchers to design/modify the data collecting hardware as well as the software which they use for experiments. No one person (or even group) has all of this knowledge – But Universities are full of expertise – where there is an idea, there is most certainly a way.