Presentation Transcript

1. virtual techdays Data grail: Data Market on Windows Azure Sudhindra Kovalam │ Developer, Icertis Inc.
2. INDIA │ 9-11 February2011 virtual techdays An Introduction to Windows Azure Marketplace DataMarket V1 Why is Data important & Why would I Need something like DataMarket in the first place? Ok Cool, What sorts of data is available on the DataMarket Developing Applications on DataMarket Using Power Pivot plugin for Microsoft Excel Developing simple applications using Data Market Inside Windows Azure DataMarket Some more details about DataMarket DataMarket internals Wrap-up S E S S I O N A G E N D A
3. INDIA │ 9-11 February2011 virtual techdays Data is the new “OIL” - Gerd Leonhard Many of the decisions we take in our day to day life are based on some sort of statistical data. As an end-user, you always want to have some statistic or data to make decisions. Discoverability and reliability Many sources on the internet such as blogs, hosted documents, websites etc. Making sense out of the data and basing your decisions on that data. A N I N T R O D U C T I O N T O D AT A M A R K E T
4. virtual techdays Example : A Site Selection application for a Retail store Chain
5. W h a t’ s D a t a M a r k e t virtual techdays
6. semantics INDIA │ 9-11 February2011 virtual techdays DEVELOPING APPLICATIONS USING DATAMARKET How does DataMarket help? schema I Need Data! billing discovery hosting
7. virtual techdays DataMarketis an information marketplace for ISVs and IWs that provides both trusted public domain and premium commercial data via integrated consumption experiences and easy data discovery, exploration, and purchasing. ENTER: MARKETPLACE DATAMARKET Global reach Unified billing & provisioning platform Easy content onboarding Data security / authorization model Flexible pricing, auditing, logging Consistent, flexible , context optimized APIs - OData Single Contract – One Stop shop for data and insight Easy access to premium data Unified billing and provisioning platform Easy discovery Rich Apps to consume Data Microsoft Office, Dynamics, Bing + 3rd party ISV Applications Ability to mash up public and private data Flexible pricing – pay as you go Information Workers Content Providers ISVs and Devs
8. semantics INDIA │ 9-11 February2011 virtual techdays DataMarket DEVELOPING APPLICATIONS USING DATAMARKET How does DataMarket help? Marketplace Consumer Spending schema I Need Data! Services forContent Providers Standard Data Access (OData) billing Billing Real Estate Value Visualization discovery hosting Climate
9. Marketplace DataMarket Content Providers DataMarket Launch 45+ Content partners 60+ Data offerings 100+ Offers coming soon
10. virtual techdays Demo : Windows Azure Marketplace DataMarket Walkthrough
11. INDIA │ 9-11 February2011 virtual techdays Using PowerPivot plugin for Excel 2010 PowerPivot gives users the power to create compelling self-service BI solutions Get PowerPivotfrom http://www.powerpivot.com/download.aspx Via Managed Code Add “Service Reference” to Dataset end point to consume the dataset via code. Exploring Data Market Datasets
12. virtual techdays Demo : Using PowerPivot for Excel 2010
13. INDIA │ 9-11 February2011 virtual techdays ODATA Protocol for querying and updating data. Based on AtomPub, HTTP, JSON Provides access to information from a variety of applications, services, and stores. Released under the Open Specification Promise. Visit www.odata.org for more details. ODATA Clients Existing - Java, PHP, AJAX, .NET, Silverlight, Excel New – iPhone, Windows Phone 7 CTP, Linqpad, etc. Developing Applications using Data Market ODATA
14. INDIA │ 9-11 February2011 virtual techdays Key Features of DataMarket APIs The Metadata service document is exposed. Useful in building a generic application which would work across multiple DataMarket datasets. Service side filtering. Fixed and Flexible Query Types (Controlled by the content provider) Fixed Querying For e.g. https://api.datamarket.azure.com/Data.ashx/adatum.com/CensusInfo/GetCensusData Proxy classes cannot be generated by doing a “Add Service reference” The proxy class file is available on the dataset Details page after the subscription is purchasedor subscribed to. Flexible Querying For e.g. https://api.datamarket.azure.com/Data.ashx/thephone-company.com/accountbalance/GetAccountBalance?phonenumber=5551234567 We can do a “Add Service reference for such datasets. Developing Applications using Data Market Key Features of Azure DataMarket
15. virtual techdays Demo : Consuming a DataMarket dataset via Code
16. INDIA │ 9-11 February2011 virtual techdays DataMarket infrastructure Basic Building Blocks Web roles for exposed datasets (web services / public DataMarket APIs) Web roles for marketplace website (ASP.NET) Public APIs Windows Azure compute infrastructure (VMs) SQL Azure elastic databases Windows Azure AppFabric Access Control System (ACS)
17. INDIA │ 9-11 February2011 virtual techdays Dataset Processing Requests Prioritized queue Dataset worker Work dispatcher Public API endpoints Dataset Prioritized queue Dataset worker . . . Dataset Prioritized queue Dataset worker
18. INDIA │ 9-11 February2011 virtual techdays Prioritized queue Data worker/ Mapping layer Caching requests Data in database Caching layer Data as web service In-memory cache Distributed cache (Windows Azure Tables)
19. INDIA │ 9-11 February2011 virtual techdays Data in database SQL Azure connector Connecting to Content Provider’s Data SQL Data worker / Mapping layer Web service connector Data source connector Caching layer OData Data as web service REST / SOAP 3rd party cloud connector REST / SOAP Data in 3rd party cloud Mapping store CSDL
20. INDIA │ 9-11 February2011 virtual techdays System data processing pipeline Partitioning function Scaling Out User DBs Partition 2 Database access management System database Part1 System database Part2 System database Part3
21. INDIA │ 9-11 February2011 Data in database Prioritized queue Data worker/ Mapping layer virtual techdays Discover issues before customer! Build monitoring as Windows Azure web role Deployed to multiple data centers Executes sample queries on regular base Stores result in Windows Azure Table Logs transferred on a regular basis to SQL Azure Reporting services to create reports Alerting on condition matching Monitoring Data as web service
22. INDIA │ 9-11 February2011 virtual techdays Data processing pipeline Problem: how to secure 3rd party connection information? Solution: Certificates! Windows Azure has dedicated certificate pipeline Only small number of people need access to them Connection information encrypted with final key during publication process Certificates to decrypt connection information on the fly Connecting to data sources Content provider information Securing High value data Crypto component Data as web service Data in database
23. INDIA │ 9-11 February2011 virtual techdays SQL Azure The Big Picture Public API endpoints Request queues / processing Data connector / data mapping Data in 3rd party cloud OData Data as web service Billing Logging and monitoring Caching Database access management System databases
24. INDIA │ 9-11 February2011 virtual techdays Enrolling as a Data consumer? Visit https://datamarket.azure.com Enrolling as a Content Provider You are in control of the data, price,and terms Easily onboard data from SQL Azure or existing web services Receive periodic reports and payouts Get in touch with the Data Market On Boarding team : DallasBD@microsoft.com How Do I Enroll
25. INDIA │ 9-11 February2011 virtual techdays Questions? Use the “Ask a Question” button on the VTD site Or use the “Q & A Panel” on the Live Meeting console. Contact Details Blog : http://sudheerkovalam.wordpress.com Email: sudhindra.kovalam@icertis.com
26. virtual techdays Thank you