Microsoft StreamInsight Introduction to Complex Event Processing with SQL Server 2008 R2 StreamInsight

1. DBI303 Microsoft StreamInsightIntroduction to Complex Event Processing with SQL Server 2008 R2 StreamInsight Torsten Grabs Lead Program Manager Microsoft Corp.

2. Understanding Streaming Data (1) • Question: “how many red cars are in the parking lot”. • Answering with a relational database: • Walk out to the parking lot. • Count vehicles that are • Red • Cars SELECT COUNT(*) FROM ParkingLot WHERE type = ‘AUTO’ AND color = ‘RED’

3. Understanding Streaming Data (2) • What about: “How many red cars took the I-80 interchange to San Francisco in the last hour”? • Answering with a relational database: • Pull over and park all vehicles in a lot,keeping them there until the end of the hour. • At the end of the hour, count vehicles that are in the lot. • Then deliver the answer Doesn’t seem like a great solution…

4. Understanding Streaming Data (3) • Different kinds of questions require different ways of answering them. • The last questions we looked at are best answered with a stream data processing engine, or complex event processing engine • How would a streaming engine do the processing for this scenario? • Stand by the freeway, count red cars as they pass by. • Keep updating the answer internally, keep delivering the answer as needed by the consumers. This is the streaming data paradigm in a nutshell – ask questions about data in flight.

5. What is StreamInsight • Microsoft’s platform to build applications for streaming data • Continuous and incremental processing • High throughput, low latency • Event-driven computation • Declarative query language (LINQ) to formulate queries, rules, and patterns • Adapter model to interoperate with all kinds of data sources and consumers • Diagnostic interface for management and administration • Extensibility model • Needs a SQL Server 2008 R2 License • Datacenter, Standard, Enterprise, Web • Evaluation, Developer

6. The Value of Timely Analytics $ value of analytics Web Analytics – Ad placement, Financial Services, Smart Grids, Monitoring – Systems mgmt, Health Care, Manufacturing, etc. Forecasting in Enterprises Historical Trend Analysis Time of interest Present

7. StreamInsight and the Microsoft Platform Visualization Distribution Processing Sources Caching Refresh (Push) Data Bus Operational Analytics Operational Dashboard (Ticking - Snapshot) Message Bus Cache Devices, Sensors Microsoft StreamInsight Reference Data Refresh (Push) Automated Decisions Reporting Dashboard (Refreshed) Web servers Relational Database ETL Static Reports Re-compute (Pull) Intra-Day Cubes Stock tickers & News feeds ETL Service Broker Historic Cubes Mining, Validation, “What-If” Scenarios

8. Event-Driven Applications Event Analytical results need to reflect important changes in business reality immediately and enable responses to them with minimal latency request output stream input stream response

9. Latency Scenarios for Event-Driven Applications Relational Database Applications StreamInsight Target Scenarios Operational Analytics Applications, e.g., Logistics, etc. Data Warehousing Applications Web Analytics Applications Manufacturing Applications Financial trading Applications Monitoring Applications Aggregate Data Rate (Events/sec.)

10. StreamInsight Platform StreamInsightApplication Development StreamInsight Application at Runtime Event sources Event targets Input Adapters Output Adapters StreamInsight Engine Devices, Sensors Pagers & Monitoring devices Standing Queries KPI Dashboards, SharePoint UI Web servers Query Logic Query Logic Trading stations Event stores & Databases Query Logic Event stores & Databases Stock ticker, news feeds

11. Virtuous Cycle: Monitor, Manage, Mine

12. Example Scenarios • Power Utilities: • Energy consumption • Outages • Smart grids • 100,000 events/sec • Manufacturing: • Sensor on plant floor • React through device controllers • Aggregated data • 10,000 events/sec • Web Analytics: • Click-stream data • Online customer behavior • Page layout • 100,000 events /sec • Financial Services: • Stock & news feeds • Algorithmic trading • Patterns over time • Super-low latency • 100,000 events /sec Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Data Stream Data Stream Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Stream Data Store & Archive Asset Specs & Parameters Event Processing Engine • Threshold queries • Event correlation from multiple sources • Pattern queries Lookup

13. Vertical: Financial Services • Scenario: Real-time RiskContinuous insight into market conditions and risk exposure • Continuous low-latency market monitoring • Manage risks across traders and per desk with aggregate and individual thresholds • StreamInsight advantage: • Implement risk monitoring declaratively in LINQ • Monitor, detect and notify risk exposure in near real-time in an event-driven way • Risk models can be re-deployed for back-testing over historical data

14. Market Monitoring demo

15. Demo Scenario: Market Monitor Real-time Dashboard Push StreamInsight Real-time/historical mesh-ups • Market Feed: • MSFT • IBM • etc. Push Grouping Aggregation Output Adapters Input Adapters Pull Push Analysis over historical data Pull

16. Microsoft StreamInsight 1.2 announcement

17. Microsoft StreamInsight 1.2 • Next release of Microsoft StreamInsight for on-premise deployments • Main benefits of the 1.2 release • Resiliency against downtime with new check-pointing capability • Support for predictive modeling and pattern matching through user-defined stream concept • Improved administrator experience with performance counters • Improved development experience with support for nested event type structures and many additional LINQ statements. • Availability of English RTM version planned for June 2011 • Licensed through SQL Server 2008 R2

18. Vertical: Web Analytics • Scenario: Real-time Behavioral Targeting • Continuously analyze online behavior per user • Identify relevant content before the next click • Define content behind next click based on detected online behavior • StreamInsight advantage: • Scale to millions of concurrent online users • Event-driven processing provides immediate insight • Web logs no longer processed offline in batches • Decisions are being made automatically as events arrive • Correlate user behavior across your web farms and applications

19. Web Activity Monitoring demo

20. Demo Scenario: Web Activity Monitoring • Acquire click-stream feed from web servers • Provide dashboards for web activity • US states heat map • Top 5 sites

21. Microsoft StreamInsighton the Windows Azure Platform Project Codename “Austin” announcement

22. Project Codename “Austin” • StreamInsight complex event processing (CEP) as a cloud service • Main benefits • Bring cloud benefits such as elastic scale to StreamInsight • Development experience in the cloud closely aligned with on-premise experience to make migrations easy • Particularly attractive where data is already born in the cloud (e.g. web analytics) • Private CTPs for early customer feedback starting now • Public CTPs planned for second half of calendar year 2011 (available thru SQL Azure Labs site)

23. Vertical: Logistics • Scenario: Fleet Management • Track current position of your vehicles • Continuously optimize routes • Optimize vehicle utilization • Schedule maintenance based on vehicle conditions • StreamInsight advantage • Gain immediate insight from sensor and event data from vehicle instrumentation • Expressive built-in algebra to calculate KPIs over your fleet • KPIs update in near-real time through event-driven processing • Easy to extend with domain-specific libraries (e.g. geo-spatial) • Include static reference data into your calculations

24. Microsoft Shuttle Tracker demo

25. Demo Scenario: Microsoft Shuttle Tracker • Plot current position for Redmond campus shuttles • Track specific shuttles • Identify when shuttles approach specific destinations • Proximity queries with SQL Spatial Libraries

26. Vertical: Power Utilities • Scenario: Smart grid • Instrument households with smart power meters • Continuous, up-to-date insight into your grid, including generation, distribution, and demand • StreamInsight advantage • Event-driven monitoring of the power grid (push instead of pull) • Scales to smart grids requirements • Scale to millions of meters • Hundreds of thousands of meter readings per second • Write validation, editing, estimation (VEE) rules declaratively in LINQ • Scale to the high data volumes expected in smart grids • React in almost real-time to changing grid conditions to avoid power outages

27. Vertical: Retail (Online and Traditional) • Scenario: Real-Time Coupon • Provide most relevant/appealing coupon upon checkout • Maximize expected individual customer revenue • Correlate current sales transaction with customer purchase history • StreamInsight advantage • Track current market basket as a real-time stream • Use StreamInsight to correlate current market basket with purchase history • Event-driven: Checkout request triggers the coupon calculation • Easily scale to internet retail with millions of concurrent sessions

28. Technology: StreamInsight Highlights • Main memory based Temporal Query Processing Engine • Declarative querying of multiple event streams • Windowing, relational and scale-out query primitives (operators) • Extensibility framework for custom operators and aggregates • Flexible hosting and deployment • Windows Service, Standalone Process, Embedded Server, Azure* • Switch from querying of historical or real-time data via configuration changes • Data connectivity via IObservable/IEnumerable or StreamInsight adapter (input/output, push/pull) • Platform Characteristics • .NET 3.5 SP1, 4.0 and CLR for the runtime • LINQ for Query Language surface • C# for apps development – Adapter, Object Model, Diagnostic APIs • Visual Studio for developer productivity (IntelliSense) • Event Flow Debugger

29. Technology: LINQ Query Examples LINQ Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 on e1.ID equals e2.ID where e1.f2 == “foo” select new { e1.f1, e2.f4 }; Filter Project &Aggregate Project Window Grouping Join LINQ Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i intoSubStream fromwin inSubStream.HoppingWindow( FiveMinutes,ThreeSeconds) selectnew { i = SubStream.Key, a = win.Avg(e => e.f) };

30. Technology: StreamInsight Deployment Alternatives Stream-Insight CEP for lightweight processing and filtering Stream-Insight CEP for aggregation and correlation Stream-Insight CEP for complex analytics including historical data • Event processing engines are deployed at multiple places on different scales: • At the edgeclose to the data source • In the mid-tierconsolidate related data sources • In the data centerhistorical archive, mining, large scale correlation Web servers Sensors Stream-Insight Stream-Insight Feeds Devices Stream-Insight Stream-Insight Stream-Insight Stream-Insight Stream-Insight Stream-Insight Complex Analytics & Mining

31. SQL Server 2008 R2 Capabilities by Edition Parallel Data Warehouse Standard Datacenter Enterprise Workload

32. StreamInsight Solutions & Partners

33. StreamInsight Platform: Recap Development experience with .NET, C#, LINQ and Visual Studio 2008 and 2010 StreamInsightApplication Development CEP platform from Microsoft to build event-driven applications StreamInsight Application at Runtime Event sources Event targets Input Adapters Output Adapters StreamInsight Engine Devices, Sensors Support for .NET sequences as sources and sinks;Flexible adapter SDK to connect to other event sources and sinks Pagers & Monitoring devices The CEP platform does the heavy lifting for you to deal with temporal characteristics of event stream data Standing Queries Event-driven applications are fundamentally different from traditional database applications: queries are continuous, consume and produce streams, and compute results incrementally KPI Dashboards, SharePoint UI Web servers Query Logic Query Logic Trading stations Event stores & Databases Query Logic Event stores & Databases Stock ticker, news feeds

34. For More Information • StreamInsight product main page • StreamInsightblog • Hitchhiker’s guide to query writing • Latest version: StreamInsight 1.1 with support for .NET 4.0 collections– Blog post with download location • StreamInsightMSDN documentation • StreamInsightsamples

35. Required Slide Speakers, please list the Breakout Sessions, Interactive Discussions, Labs, Demo Stations and Certification Exam that relate to your session. Also indicate when they can find you staffing in the TLC. Related Content DEV340 Tackle the Complexity of Async Calls in Microsoft Silverlight and WPF Clients • SQL Server Programmability Booth • Find Me Later At… … the Booth

36. Resources • Connect. Share. Discuss. http://northamerica.msteched.com Learning • Sessions On-Demand & Community • Microsoft Certification & Training Resources www.microsoft.com/teched www.microsoft.com/learning • Resources for IT Professionals • Resources for Developers http://microsoft.com/technet http://microsoft.com/msdn

37. Complete an evaluation on CommNet and enter to win!

39. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.