fjording the stream an architecture for queries over streaming sensor data n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Fjording The Stream An Architecture for Queries over Streaming Sensor Data PowerPoint Presentation
Download Presentation
Fjording The Stream An Architecture for Queries over Streaming Sensor Data

Loading in 2 Seconds...

play fullscreen
1 / 31
judah

Fjording The Stream An Architecture for Queries over Streaming Sensor Data - PowerPoint PPT Presentation

118 Views
Download Presentation
Fjording The Stream An Architecture for Queries over Streaming Sensor Data
An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Fjording The StreamAn Architecture for Queries over Streaming Sensor Data Samuel Madden, Michael Franklin UC Berkeley

  2. Introduction • Telegraph Sensor Query Processing Architecture • Fjords • Enable push and pull in query plans • Operators for streaming data • Sensor proxy • Sensor - Query Mediator User Proxy Web Sensors Fjording ICDE

  3. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  4. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  5. Sensor Challenges • Battery Powered • 2AA Cells (2800 mAh; 1.01x104 J), Coin Cell (100mAh) • Communication Dominates Power Cost • Can be exhausted - tens to hundreds of MBs of data / sensor • Wireless • High loss rates (20% @ 5meters , typical) • Low bandwith (10kbps) • Near Real Time • Streaming Data • Pushed at (user defined) regular intervals TinyOS “Mote” Fjording ICDE

  6. Requirements of Sensor Query Processing • Power Sensitivity • Tolerance to unbounded streams of data • Tolerance to push • Tolerance to intermittent, lossy connections • Tolerance to failed sensors Fjording ICDE

  7. Traffic Scenario • CA Department of Transportation (CalTrans) has sensors all over bay area freeways • Inductive loop sensors give speed, flow, vehicle size • Motes could collect this data cheaply • Many possible queries • Commuters want to find congestion • California Highway Patrol (CHP) wants to find accidents Fjording ICDE

  8. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  9. Fjords • Query plan implementation • Useful for streams and distributed environments • Combine push (streaming) data and pull (static) data • E.g. traffic sensors with CHP accident reports Fjording ICDE

  10. Push vs. Pull • Problem: Need an API to combine push & pull • Operators (e.g. join) data-direction agnostic • Push vs. pull implemented in queues (connectors) • Contrast with • Iterator model • Exchange operator • Allows arbitrary combinations of push and pull Fjording ICDE

  11. get() process() Pull Example Pull Queue Operator parent, child; Tuple get() { Tuple t = null; while (t == null) { t = child.process(); } return t; } Operator Queue q;… Tuple process() { Tuple t = q.get(), outt = null; If (t != null) { <process t> } else { … do something else … } return outt; } s • Notice: • Iterator semantics by making get() blocking • Get() can return null • Process() can return null Pull Connection Scan Fjording ICDE

  12. get() Push Example Push Queue Operator parent, child; Vector v = new Vector(); Tuple get() { if (v.size() > 0) return v.removeFirst(); else return null; } Tuple enqueue(Tuple t) { v.put(t); } Operator Queue q;… Tuple process() { Tuple t = q.get(), outt = null; If (t != null) { <process t> } else { … do something else … } return outt; } Thread while(true) { Tuple t = op.process(); if (t != null) op.outq.enqueue(t); } s Push Connection Scan Fjording ICDE

  13. Fjord Example Push Push Pull Fjording ICDE

  14. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  15. Continuous Queries • Given user queries over current sensor data • Expect that many queries will be over the same data sources (e.g. traffic sensors) • Queries over current data always looking at same tuples • Those queries can share • Current tuples • Work (e.g. selections) • Sharing reduces messages, thereby power! • A new query can be “folded” into an existing query • Use old instances of scans and selections Fjording ICDE

  16. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  17. Relational Operators And Streams • Selection and Projection Apply Naturally • No Blocking Operators • Sorts and aggregates over the entire stream • Nested loops and sort-merge join • Windowed Operators • Sorts, aggregates, etc. • Online, Interactive QP Techniques • In memory symmetric hash join • Alternatives: ripple-join, Xjoin, etc. • Partial Results Fjording ICDE

  18. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  19. Query Registration Parsed Queries [sources, ops] [fields, filters, aggregates, rates] Query [tuples] Operators Sensor Proxies • Mediate between Sensors and Fjords • Push operators out to sensors • Hide query processing, knowledge of multiple queries from sensors • Hide details of sensors from query processor • Enable power-sensitivity Query Processor Fjording ICDE

  20. What runs where? • (Multi-query) Distributed Optimization Challenge: • Given set of operators, proxy must choose: • Run on sensor or, • Run on local query processor • Running on sensors saves power • Simple computations cheaper than messages • Selection, aggregation reduce communication cost • Sensors have limited resources • All queries can’t run in all sensors simultaneously • Limited state precludes big joins or lots of groups • Queries share operators • Operators vary in selectivity Fjording ICDE

  21. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  22. accident = true Index NL Join Building a Query • How to translate a declarative query into a Fjord? • Just like traditional query processing, except: • Branches originating in sensors connected by push connectors • Sensor proxy handles scans, selections over sensors • Proxy delivers tuples from sensors • Proxy pushes-down selections transparently • Output of join is push if one or both inputs is push • Join carefully chosen Pull: Data Request Data Request Speed < 30 Push Push Data Fjording ICDE

  23. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  24. Loc  {userLocs} Query & Fjord • Simple Test Query: SELECT AVG(s.speed, w) FROM sensors AS s WHERE s.loc in {userLocs} Telegraph Server Average BHL Server Fjording ICDE

  25. Loc  {userLocs} Loc  {userLocs} Loc  {userLocs} Multiple Queries in a Fjord Telegraph Server BHL Server Fjording ICDE

  26. Fjord Performance (In Telegraph) Fjording ICDE

  27. w t : Vehicle Covers S t : Vehicle Covers S S and 0 u 1 u d w w t : Vehicle Covers S t : Vehicle Covers Neither Sensor 2 d 3 Sensor Proxy for Traffic • Measure benefit of pushing computation into sensors; in this case, vehicle identification. • Simple “aggregation” dramatically reduces power costs Fjording ICDE

  28. Pushing Aggregates Saves Power Atmel (TinyOS CPU) Simulator 100 samples / sec 5 vehicles / sec 7x power savings Fjording ICDE

  29. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  30. Related Work • Cougar • Interactive & Adaptive Query Processors • Tukwila, Xjoin, Eddy, CONTROL • Continuous Query Processors • NiagaraCQ, Xfilter, CACQ • Directed Diffusion • Volcano / Exchange Operator • Temporal Databases Fjording ICDE

  31. Conclusions & Future Work • Required for sensor query processing: • Fjords : API for combining push & pull • Sensor Proxy: Mediator between sensors and QP • Streaming Operators • Big benefit from pushing selections, aggregates into network • Combining multiple queries is a win • Extensions & Future Work: • Multi-query optimization & adaptivity (CACQ, SIGMOD 2002) • Push down selections & aggregates (Submitted to VLDB 2002) • Sensor proxy policies Fjording ICDE