1 / 14

Enhancing Seismic Data Processing with ANF Detection Network and SQLstream Integration

The ANF Detection Network project leverages cloud computing and SQLstream technology to operate on seismic data in real-time. By employing elastic processing, the system manages data from the Array Network Facility, enhancing detection capabilities and improving overall performance. The architecture includes an Application Controller and Agent that facilitate communication and management of SQLstream instances. This document outlines the goals, deployment details, and future objectives, such as data preservation and improved monitoring for a more robust processing system.

sheera
Download Presentation

Enhancing Seismic Data Processing with ANF Detection Network and SQLstream Integration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ANF Detection Network Dave Foster dfoster@asascience.com Guy de Wardenergdewardener@asascience.com

  2. Resources • Confluence page: https://confluence.oceanobservatories.org/display/CIDev/ANF-SQLstream+Integration • ANF Detection Network code: https://github.com/daf/anf-detection-network

  3. Introduction / Goals • Using elastic processing (cloud computing) to operate on seismic data • Drive development and test ION (https://confluence.oceanobservatories.org/display/CIDev/Home) • Data from Array Network Facility (http://anf.ucsd.edu/) • Uses SQLstream (http://sqlstream.com/) for processing for real time processing

  4. (con’t) • Continuation of POC/Prototype (Aug 2009, https://confluence.oceanobservatories.org/display/CIDev/SQLstream+Prototype)

  5. Architecture

  6. (con’t) • ANF Application Controller • Part of four op units in a deployment • Listens for instrument announcements, partitions those instruments into AMQP queues, requests SQLstream worker op units be created • ANF Application Agent • Runs on SQLstream worker op unit • Manages SQLstream instances, communicates status with Application Controller

  7. AppAgent • Installs/configures SQLstream on demand from AppController • Allows us to use common base image • More than one SQLstream instance (listening to different queues) can be run on a worker op unit • Downsides: time spent transferring installer from S3, time spent installing SQLstream

  8. SQLstream • Listens to AMQP queue assigned to it by App Controller • Transmits detection events over AMQP topic “anf.detections”

  9. Deployment • Uses CEI developed tools cloudinit.d, epumgmt • cloudinit.d: https://github.com/nimbusproject/cloudinit.d • epumgmt: https://github.com/nimbusproject/epumgmt • Actual startup commands: • cloudinitd boot launch-plans/anf/main.conf -n anf -v -v -f /tmp/anf.log • epumgmt.sh -a find-workers -n anf --wholerun • epumgmt.sh -a logfetch -n anf --wholerun

  10. Demo • Live system • 30 stations • Captured Data • 10 stations, showing events

  11. Metrics Instrument Announce ~ 7:20 startup

  12. (con’t) SQLstream running 4:58 36s 12.5s 44.47s 175.40s (2:55)

  13. Future Goals • Data Preservation • Detections, possibly input events • Service Hardening • Distributed, stateless processes • Replacement for instrument announcement • True load balancing • Need more metrics with live data

  14. (con’t) • Monitoring • Graphical dashboard showing SQLstream op units, queues/bindings, consumption rates

More Related