1 / 44

Skyalert: Rapid Transients for you and your Robot

Skyalert: Rapid Transients for you and your Robot. Roy Williams LIGO Laboratory, Caltech Andrew Drake, Ashish Mahabal Astronomy, Caltech. Supported by US National Science Foundation OCI-0915473. http://lib.skyalert.org/skyalert/Guide2Skyalert-1.3.pdf. Skyalert is ….

zenia
Download Presentation

Skyalert: Rapid Transients for you and your Robot

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Skyalert: Rapid Transients for you and your Robot Roy Williams LIGO Laboratory, Caltech Andrew Drake, Ashish Mahabal Astronomy, Caltech Supported by US National Science Foundation OCI-0915473

  2. http://lib.skyalert.org/skyalert/Guide2Skyalert-1.3.pdf Skyalert is …. A clearinghouse and repository of information about astronomical transients, each described by a collection of VOEvent packets that may be multiply authored. The components of Skyalert are: • A web-based event broker, allowing subscription so that information about transients can be delivered to users and their telescopes immediately upon receipt. • A web-based authoring system, so that authenticated users can inject events direct from automated discovery pipelines, or fill in web forms, that may be delivered rapidly to others. • An event repository, storing all events that come through the broker, and allowing bulk queries and drill-down. • A ‘click or code’ paradigm that allows people web based access and machines web service access. • A way to browse recent and past transients, as tables, multi-layered web pages, or with popular astronomical software. • A development platform for building real-time decision rules about transients, and for mining the repository. • Open-source software to allow local implementations as well as the web-based application.

  3. http://lib.skyalert.org/skyalert/ Installations • Production: Skyalert.org at Caltech • Test: Betelgeuse.ligo.caltech.edu:8000 • IUCAA, India • LSST, Tucson, Arizona • AMON, Penn State, Pennsylvania • GAIA, Cambridge UK

  4. http://skyalert.org Skyalert Authors Subscribers AstronomersAmateursStudents GCN Broker annotation from archives Microlensing Optical transients Radio transients X-ray transients Gamma transients skyalert.org Events and annotation disseminated to subscribers in real time with intelligence Follow-up Scheduler Telescope Telescope Telescope

  5. http://ivoa.net/Documents/VOEvent/ http://hotwireduniverse.org http://www.skyalert.org/s/sampleEvent.html VOEvent is the common language

  6. http://ivoa.net/Documents/VOEvent/ http://hotwireduniverse.org VOEvent http://www.skyalert.org/s/sampleEvent.html • Who – author’s provenance • What – author’s data packet • WhereWhen – targeting in spacetime • Why – scientific characterization • Citation – related events • IVORN – unique identifier <voe:VOEvent xsi:schemaLocation="http://www.ivoa.net/xml/VOEvent/v2.0 /Users/roywilliams/Documents/VOEventLib/schema/VOEvent-v2.0.xsd" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:voe="http://www.ivoa.net/xml/VOEvent/v2.0" version="2.0" ivorn="ivo://ldas-pcdev1.ligo.caltech.edu/VOEvent#a8" role="test"> <Description>Candidate galaxies for LIGO trigger event</Description> <Reference uri="http://ldas-pcdev1.ligo.caltech.edju/VOEventRepository/1"/> <Citations> <EventIVORN cite="followup">ivo://ldas-pcdev1.ligo.caltech.edu/VOEvent#a7</EventIVORN> </Citations> <Who> <Author> <title>LSCVirgo</title> <contactName>Ed Daw</contactName> <contactEmail>e.daw@shef.ac.uk</contactEmail> <contactPhone>+44 114 268 6895</contactPhone> </Author> </Who> <What> <Param name="EVETut_ID" value="012345678" ucd="meta.id"> <Description>use the gps second as an event ID </Description> </Param> <Param name="SNR" value="2.4" ucd="stat.snr"> <Description>Significance of the trigger</Description> </Param> <Param name="UTC" value="2005-04-15T23:59:59" ucd="time.epoch"> <Description>Time of the trigger</Description> </Param> <Param name="NO_OF_GALAXIES" value="3" ucd="meta.number"> <Description>Number of candidate source galaxies in this VOEvent</Description> </Param> <Table> <Description>Candidate sources for the LSC/Virgo trigger</Description> <Field name="Source rank" ucd="meta.code"> <Description>this integer is 1 for the most likely source, 2 for 2nd, etc</Description> </Field> <Field name="GWGC_CATALOGUE_NUMBER" ucd="meta.number"> <Description>this is the integer ID of the galaxy in Darren's catalogue</Description>

  7. http://www.skyalert.org/submit/ VOEvent validation Choose herefor validation

  8. http://lib.skyalert.org/VOEventLib/ VOEventLib • Python Library for VOEvent • Read, modify, create, write VOEvents • lib.skyalert.org/VOEventLib • Code built automatically from XML schema • GenerateDS package • Thanks Dave Kuhlman! from VOEventLib.VOEvent import * from VOEventLib.Vutil import * import sys # get filename with event to be modified # parse the event from the file name v = parse(filename) # set the ivorn to something else v.set_ivorn('ivo://silly/billy#iuw6e7F72ufh') # set the author v.get_Who().get_Author().set_contactName(['Mickey Mouse']) # look for a specific param in the event to change param = findParam(v, '', 'magnitude') if param: val = paramValue(param) param.set_value("16.45") xml = stringVOEvent(v) print xml

  9. Who uses VOEvent • IAU/CBAT, AAVSO • GAIA • CRTS, CSS asteroids, MOA, OGLE • SWIFT, Fermi, Integral, Agile, etc (GCN translation) • LIGO, LOFAR • LSST? • amateurs (DC3.com)

  10. http://www.skyalert.org/ Event timeline click event for drilldown

  11. Worldwide Telescopeview of all recent events CLICK

  12. Skyalert Key Concepts • Identifier • IVORN • Stream (class) IVORN inside event (instance) IVORN • Stream • Event semantics defined by author • What does Burst_SOD mean in the Fermi stream? • Like a data dictionary, a schema, common metadata • Portfolio • Event + associated data • As collection of VOEvents with the observation first • Alert/Select • Define what is ‘interesting’ as Python expression on Params • eg CRTS[“magnitude”] < 17 • Get message, run annotator, run telescope • Actually … everything is an Alert Fermi CRTS3 PIoftheSKY ?

  13. http://www.ivoa.net/Documents/latest/IDs.html Concept: Identifier== IVORN • Can identify a stream: • ivo://moa.massey/transient_alerts • Can identify an event from that stream: • ivo://moa.massey/transient_alerts#gb10-R-8-16933_1  The event IVORN contains the stream IVORN

  14. Concept: Streamas Event TemplateEvent author defines parameters and meaning in advance. stream event made byrobot system at night made byperson in daytime

  15. http://www.skyalert.org/streams/54/ Stream Definition Name, IVORN, owner Description Overview Template See events, delete tests Owner view Public view Parameters that can be used in the overview Sample event

  16. http://www.skyalert.org/streams/ Skyalert Stream Registry... will be VO registry

  17. Concept: Portfolio ==“data holdings about a transient” Multiple authors Primary event plus secondary data: follow-up, archives, data-mining results etc In VOEvent there are <Citation> tags

  18. http://www.skyalert.org/events/46556/ Portfolio skins XML Params Overview

  19. http://www.skyalert.org/events/33547/ CRTS portfolio Later CRTS observation CRTS observation Telescopic followup Is it in the archives yes: NED and NVSS Classification as blazar outburst

  20. http://www.skyalert.org/events/49299/ SWIFT portfolio Notices from the satellite Circulars from the humans

  21. What is an Alert? • Selects “interesting” events • For rapid messaging • For response to query • Default action is to send email, some others: • addconstellation • alert_email • coincidence • csannotator • post2skyalert • sdssannotator • senddakota ?

  22. http://www.skyalert.org/s/annotator.html Annotators • When triggered, annotator builds event, appends to portfolio • Annotator can be local or remote • Example • For each event, find nearest galaxy in SDSS

  23. http://pipes.yahoo.com/pipes/ Every Alert is a FeedAtom and RSS

  24. Skyalert Overview Messaging Email Jabber Dakota etc Streams Authoring Alert Events Annotation Browsing JSON services Feeds Portfolio Replication GCN Email Jabber Dakota etc ? ? ? = http POST

  25. Users, Groups, Streams Auth and Auth Users register Get email & respond Each stream has User who is owner Users can join Groups Groups can have permission to view streams

  26. Tutorials • Tut1: Stream, Sample event, Authoring • Tut2: Authoring with Code • Tut3: Alert, Annotation • Tut4: Mining Please use test machine https://betelgeuse.ligo.caltech.edu:8000/

  27. https://betelgeuse.ligo.caltech.edu:8000/streams/54/ Tut1: Stream Look at “Test” stream Anonymous and Logged in Play with template Look at bottom for “Sample Event”

  28. https://betelgeuse.ligo.caltech.edu:8000/streams/sampleEvent/54/https://betelgeuse.ligo.caltech.edu:8000/streams/sampleEvent/54/ Tut1: Sample Copy sample and Edit Change IVORN ivorn="ivo://testing/test#5551" Change RA, Dec Change ISOtime Change Params Go to Authoring page Keep this on your desktop

  29. https://betelgeuse.ligo.caltech.edu:8000/submit/ Tut1: Validation and Authoring Select “Validate XML” Click to Go Now authoring … Enter stream name “Test” Enter username/pass Click to Go

  30. Tut1: Authoring Event successfully saved Portfolio visible

  31. http://www.skyalert.org/s/authorcode.html Tut2: Authoring by code import urllib dict = {} # the server that will handle the submit request url = "https://betelgeuse.ligo.caltech.edu:8000/submit/" # choose 'dryrun' for validation and 'author' for authoring dict['checker'] = 'dryrun' # for command line, we want plain text output, not HTML dict['plainResponse'] = 'on' # uncomment these for authoring #dict['checker'] = 'author' # Skyalert username and password #dict['username'] = username #dict['password'] = password # This is the short name for the stream, # must match credentials and event! #dict['streamName'] = 'myEventStream' # Should alerts be run once the event is ingested? #dict['doRules'] = 'on' # open a file for the XML dict['xmlText'] = open('sample.xml').read() # Now send it off and print the result params = urllib.urlencode(dict) f = urllib.urlopen(url, params) result = f.read() print result • Very simple code • http gets through firewalls • Standard Python/Perl etc • Can also author with • Jabber, Dakota, Email Change server Put in username/password Make XML file from desktop must change IVORN

  32. https://betelgeuse.ligo.caltech.edu:8000/rules/newrule/ Tut3: Building an Alert Log in to build your own alerts and feeds Select the “Test” stream ?

  33. https://betelgeuse.ligo.caltech.edu:8000/rules/337/ Tut3: Building an Alert Select the “Test” stream Give it a name Choose action as csannotator Make trigger expression Can use red dots below Or type Python Click Save ?

  34. Tut3: Authoring with Consequences Run with Alerts Have changed RA and Dec of event to 194.95 and 27.98

  35. Tut3: Annotator Ran Alerts run on Test events Annotator fetches data from VO archives

  36. https://betelgeuse.ligo.caltech.edu:8000/events/49663/ Tut3: Discover Coma Cluster Annotator (VO archives) reveals Coma Cluster from NED

  37. Tut4: Mining the Events Alert is a query Use JSON to get query result -- page/resumption Then JSON to get portfolio detail

  38. http://www.skyalert.org/events/jtable/285/ Tut4: Select • List the portfolios that satisfy an alert with JSON return: /jtable/ { "rule": "285", "query": "trigger expression", "next": resumption URL, "list": [ [ "http://....", "ivo://....", "2011-09-19 02:12:23" ], from django.utils import simplejson import urllib sk = ‘http://skyalert.org’ # CRTS and CRTSCircular["First"]["eventClass"] == "Supernova" query = "285" jtableurl = "%s/events/jtable/%s/" % (sk, query) while 1: jsontable = urllib.urlopen(jtableurl).read() pytable = simplejson.loads(jsontable) list = pytable["list"] print "Found %d portfolios" % len(list) for triple in list: ivorn = triple[1] print ivorn

  39. http://www.skyalert.org/resolve/ Tut4: Resolve • Resolve an IVORN to the portfolio: /resolve/ { "url": "http://skyalert.org/events/50425", "portfolio": { "ivo://.....": { "Group1": { "Param1": "...", "Param2": "….", # get magnitude from a CRTS event sk = "https://betelgeuse.ligo.caltech.edu:8000" dict = {‘ivorn’:ivorn, ‘JSON’:’on’} params = urllib.urlencode(dict) resolveurl = "%s/resolve/" % sk jsondetail = urllib.urlopen(resolveurl, params).read() pydetail = simplejson.loads(jsondetail) primaryevent = pydetail["portfolio"][ivorn] mag = primaryevent["FirstDetectionparams"]["magnitude"]

  40. Tut4: Mining the Events from django.utils import simplejson import urllib #sk = "http://skyalert.org" sk = “https://betelgeuse.ligo.caltech.edu:8000” # CRTS and CRTSCircular["First"]["eventClass"] == "Supernova" query = "285" jtableurl = "%s/events/jtable/%s/" % (sk, query) while 1: jsontable = urllib.urlopen(jtableurl).read() pytable = simplejson.loads(jsontable) list = pytable["list"] print "Found %d portfolios" % len(list) for pf in list: ivorn = pf[1] dict = {} dict['ivorn'] = ivorn dict['JSON'] = 'on' params = urllib.urlencode(dict) resolveurl = "%s/resolve/" % sk jsondetail = urllib.urlopen(resolveurl, params).read() pydetail = simplejson.loads(jsondetail) # ivorn of the primary event of the portfolio should be here crtsevent = pydetail["portfolio"][ivorn] mag = crtsevent["FirstDetectionparams"]["magnitude"] tok = ivorn.split('#') localivorn = tok[1] date = crtsevent[""]["ISOtime"] print localivorn, date, mag jtableurl = jsonresult["next"] if jtableurl == "finished": break Magnitudes of CRTS supernovae 1107030071174105491 2011-07-03T08:36:01 18.935400 1107030091124153498 2011-07-03T08:01:30 18.756399 1107031180794128821 2011-07-03T04:39:47 20.169800 1107031070764142675 2011-07-03T04:07:39 18.166100 1107021180094118167 2011-07-02T10:46:06 18.601400 1107021041264109182 2011-07-02T09:31:06 15.879600 1107011150104107901 2011-07-01T10:47:36 18.279699 1107011120104119909 2011-07-01T10:46:15 17.874800 1107010040834139996 2011-07-01T06:17:32 18.483601 1107011040674116917 2011-07-01T04:13:08 19.630899

  41. http://webstats.skyalert.org Skyalert Statistics 61 gmail.com 24 aol.com 15 yahoo.com 7 hotmail.com 6 mail15.com 6 astro.caltech.edu 5 camerabuy.info 5 cacr.caltech.edu 4 caltech.edu 3 lsst.org 2 yahoo.com.br 2 yahoo.co.uk 2 ya.ru 303 registered users

  42. Skyalert Statistics2724 emails sent May 18 thru June 18 Which streams generated them? 910 SWIFT 785 CRTS 340 CBAT 204 CRTSCircular 192 MOA 156 CRTS2 111 Fermi 20 AAVSO 6 HST_MCT Emails per day Who got them? 324 aam@astro.caltech.edu 215 scott@milkyway.gsfc.nasa.gov 193 s.littlefair@shef.ac.uk 186 roy@cacr.caltech.edu 178 holtz@nmsu.edu 139 vxw@capella2.gsfc.nasa.gov 130 rcthomas@lbl.gov 116 mcastro119@gmail.com 77 raderrobert@yahoo.com 77 nikhil_paramane5@rediffmail.com 77 michaeljcook@rogers.com 77 jfl0132@gmail.com 77 jeffrey.d.scargle@nasa.gov 77 cwu@bao.ac.cn full new

  43. Questions? Please register at skyalert.org

  44. Slide 2: http://lib.skyalert.org/skyalert/Guide2Skyalert-1.3.pdf Slide 3: http://lib.skyalert.org/skyalert/ Slide 4: http://www.skyalert.org Slide 5 and 6: http://ivoa.net/Documents/VOEvent/ Slide 7: http://www.skyalert.org/submit/ Slide 8: http://lib.skyalert.org/VOEventLib/ Slide 10: http://www.skyalert.org Slide 13: http://www.ivoa.net/Documents/latest/IDs.html Slide 15: http://www.skyalert.org/streams/54/ Slide 16: http://www.skyalert.org/streams/ Slide 18: http://www.skyalert.org/events/46556/ Slide 19: http://www.skyalert.org/events/33547/ Slide 20: http://www.skyalert.org/events/49299/ Slide 22: http://www.skyalert.org/s/annotator.html Slide 23: http://pipes.yahoo.com/pipes/ Slide 27: https://betelgeuse.ligo.caltech.edu:8000/streams/54/ Slide 28: https://betelgeuse.ligo.caltech.edu:8000/streams/sampleEvent/54/ Slide 29: https://betelgeuse.ligo.caltech.edu:8000/submit/ Slide 31: http://www.skyalert.org/s/authorcode.html Slide 32: https://betelgeuse.ligo.caltech.edu:8000/rules/newrule/ Slide 33: https://betelgeuse.ligo.caltech.edu:8000/rules/337/ Slide 36: https://betelgeuse.ligo.caltech.edu:8000/events/49663/ Slide 38: http://www.skyalert.org/events/jtable/285/ Slide 39: http://www.skyalert.org/resolve/ Slide 41: http://webstats.skyalert.org

More Related