1 / 40

Sybase IQ: Changing the Game for Extreme Analytics

Sybase IQ: Changing the Game for Extreme Analytics. Michael Breen Technical Director Sybase ISV/OEM Engineering March 10, 2011. The Unwired Enterprise. Our focus is to help customers realize the Unwired Enterprise . INFORMATION ANYWHERE, ANYTIME. Growing Pains.

jess
Download Presentation

Sybase IQ: Changing the Game for Extreme Analytics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Sybase IQ:Changing the Game for Extreme Analytics Michael Breen Technical Director Sybase ISV/OEM Engineering March 10, 2011

  2. The Unwired Enterprise Our focus is to help customers realize the Unwired Enterprise. INFORMATION ANYWHERE, ANYTIME

  3. Growing Pains “My organization is facing… • Data and user explosion • More users need more access to more data – volume, variety, velocity • New regulatory requirements to keep more data online longer • Traditional Technology Falling Short • Query workloads drag down operational performance – need to offload reporting and analysis • Existing reporting servers are getting too big, too unwieldy and too expensive • DBAs spending too much time optimizing queries • Increased demand for analytics and reporting • Competitive climate pushes companies toward real-time business agility—instantaneous access tobusiness intelligence & self-service reporting • Regulatory requirements increase the need for reporting and accountability • Cost and risk reduction drives closer scrutiny of non-compliance or fraud Analytics is the key to unlocking differentiation and inspired execution

  4. Introducing Sybase iq Sybase IQ turns your information into better decisions. • In an age of no excuses, a winning enterprise is one that makes the best decisions. Decision-makers must analyze business trends, predict outcomes, and revise strategies, often in a matter of seconds. • No matter: • The volume or type of information • The speed at which it’s needed • The detail to which it is examined • The number and level of decision-makers involved • Sybase IQ delivers precise, actionable answers to drive better decisions for winning enterprises. • Sybase IQ turns information into action, and enables decision-makers at all levels to produce the results that will drive your business forward. Sybase IQ Smarter Answers Now • More than 1800 customers; more than 3200 installs worldwide • Provides answers 10 – 1000X faster • Patented data compression dramatically reduces data storage requirement; cuts TCO • Only column-based solution to support full text search, in-database analytics, and federated analytics • Leader, Gartner Magic Quadrant for Data Warehouse DBMS

  5. Sybase IQ is the market leader #1 column-based analytics server • Providing decision makers with the answers they need for over 10 years • Outperforming other databases by 10 to 1,000 times • Over 3,200 unique installations in over 1,800 companies…and growing

  6. Smarter analytics with Sybase IQ In an economic downturn, companies can’t afford to bet their business intelligence on yesterday’s systems • Increasing market demands requires specialty analytics servers • Sybase IQ is #1 column-based analytics server with over 1800 customers • Times are lean; Sybase IQ is green. The world’s top-performing analytics serveris also the one easiest on your budget…and the planet!

  7. "This is an across-the-board upgrade that provides a whole new, entirely different, experience for users — especially power users." — Dan Feshbach CEO, LoanPerformance “The customers say it is incredible. They are blown away by what they can do." — Henri Asseily Chief Technology Officer and Co-founder, Shopzilla “This architecture is presented as a model for other divisions and agencies for data warehousing.." — Sam Foster, President FosterSoft, SEC contractor “Our customers benefit directly as well. The speed improves our ability to mine the data and produce results for our customers much more quickly. That helps them market more effectively and generate more business.” — Ric Elert Vice President of Engineering, comScore “Using Sybase IQ, we can deliver services we never thought possible, and which will benefit our public and private sector customers immeasurably for years to come."—Alison Mackenzie, manager of the HALOGEN project, Mott Macdonald “We are able to deliver one data warehouse for all of our applications at one-third the storage of conventional technologies, while seeing performance gains as advertised.” — Kim Ross CIO, Nielsen Media Research “We have seen remarkable improvements in productivity, and big decreases in training, support, and hardware costs." — Jeff Butler Director of Research Databases, Office of Research, U.S. Internal Revenue Service “The IBM and Sybase IQ combination offers unbeatable cost/performance and gives S&H a competitive edge in retail analytics." — Steve Ritacco CTO, S&H Solutions What Customers Say

  8. LoanPerformance Mortgage Risk Intelligence at the Speed of Light "Installation of the new Sybase IQ system took only a week and immediately delivered dramatic results. The number of queries that took more than 10 minutes was reduced from 16 percent to less than one percent." Dan Feshbach CEOLoanPerformance Challenges • Slow-running queries made TrueStandings undesirable to customers • Customer-facing, ad-hoc access via Web interface • Unable to scale number of concurrent users • Data volumes large and growing • All mortgages/loan data in United States… in the billions Business Results • Average eight times – and up to 100 times – faster than the previous system • No query, even the most complex, over 10 minutes • Implementation within a week © 2006 Sybase, Inc.

  9. Sybase IQ’s Secret Ingredient:Column-Based Architecture Conventional Database • Data is stored horizontally • Querying without indexes and views is extremely I/O intensive • Building indexes and views is a huge time and resource drain • Database footprint must be dramatically expanded to make the environment efficient for querying c1 c2 c3 c4 c5 c6 c7 c8 c9 … r1 r2 r3 r4 r5 SYBASE IQ • Data is stored vertically – Each column is stored separately • The data is the index • Retrieve only columns used in the query – Reduce System I/O dramatically • Allocate a thread for each column individually – Process the query in parallel c1 c2 c3 c4 c5 c6 c7 c8 c9 … r1 r2 r3 r4 r5

  10. VLDW CUSTOMERS Proven Data Compression Savings Sybase IQ Data Compression: saves money – saves time

  11. Sybase IQ Features/Benefits

  12. Sybase IQ Features/Benefits

  13. The Sybase IQ Advantage • 10-1000x performance improvement • Up to 90% less storage • Scalability and flexibility from multiplexing • Lowest TCO and lowest energy use How can we make these advantages work for you?

  14. Data is Stored Vertically Each column is stored separately Bit-Mapped Index Index on every column FAST ACCESS and LOAD Optimized Storage Input data is compressed Usually = 40%-70% Database smaller than input data Even with all the indexes Query Engine Retrieves Only Columns Used in the Query Reduces system I/O dramatically Average 90% less than competition Permits better data manipulation Easy to alter and manage Schema Design Not Restricted Design based on application use Flat, Star, Relational, Snowflake Any Schema Designed for Analytics: So What’s Different?

  15. Speed from a column-based architectureA Basic Query into a Growing Customer Database An online shopping service has millions of subscribers. Valentine’s Day is coming. How many male customers in California have ever purchased jewelry? Row-based Database 800 Bytes x 10M 16K Page = 500,000 I/Os Jewelry Purchases Gender State M NY Y M CA N F CT N M MA Y M CA Y – – – 10M Rows • Process large amounts of unused data • Often requires full table scan 800 Bytes/Row 10M Bits x 3 Column/8 Bits 16K Page = 234 I/Os Column-based Sybase IQ Jewelry Purchases Gender State M Y CA M N CA F Y NY M Y CA 1 1 0 1 0 1 0 1 1 1 0 1 10M Rows + + 2 = 10M Bits

  16. Conventional DBMS SummariesAggregates 1 – 2 TB Same INPUT data:“Conventional DW”is 3x-6x larger thanSybase IQ DW Indexes 0.5 – 3 TB 2.4-6 TB Base table“RAW data”no indexes 0.9 – 1.1 TB Aggr/Summ: 0 - 0.1 TB 0.25 - 0.9TB Indexes: 0.05 - 0.3 TB Base table: 0.2 - 0.5 TB Data CompressionDramatically Reduces Storage and Maintenance LOAD INPUT DATA: 1 TB Source: flat files, ETL, replication, ODS LOAD

  17. Query Results – Financial Customer Hardware: IBM 570 systemP 16 cores(8 Dual Core)

  18. Pick-n-Pay:Reporting solution for a leading grocery retailer Issues Increased data volumes and concurrent user query and reporting demands 32-processor Teradata warehouse could not provide acceptable performance at an acceptable cost Needed a highly cost-effective, scalable reporting solution that could integrate data from many operational systems Results of the Sybase IQ Implementation Reduces annual costs Improves efficiency and access to live operational data Enhances performance and query response times Integrates easily with SAP ERP System SybaseIQ Report Server Accelerator “Even in the early stages of development, IQ is already delivering improved performance in corporate reporting.” Harold Dawson, IT Director Pick ’n Pay

  19. Global Credit Card Company • Business Issues • Unable to perform advanced analysis of fraud patterns for credit card transactions with competing solutions due to performance issues, query complexity limitations • Needed solutions to handle 700-column table to describe every transaction, at least one year of transactions online • Results • Advanced fraud analysis possible for last 4 years • Over 1,600 users worldwide • 39TB in Sybase IQ • 10 billion records (last 13 months) of credit card transactions online • All fraud managers worldwide use IQ Multiplex system • Over 90% of database is fraud detection information

  20. Mem Mem Mem Mem MultiplexingIncremental Scalability and Flexibility Can start with 1 server and add CPUs and memory as needed IQ Reader Node IQ Reader Node IQ Reader Node IQ Writer Node IQ Reader Node CPU CPU CPU CPU CPU CPU CPU CPU CPU Mem Multiplexing enables you to add servers and CPUs with little or no loss of performance Mem CPU CPU Mem CPU CPU CPU Mem CPU Mem CPU Mem Mem Mem CPU Mem Fiber Channel SAN Terabytes of disk can be added to the SAN and IQ will manage it efficiently • Each node works independent of the other nodes • No competition for system resources between nodes • Match the hardware to the workload • Provides a high-availability solution • Not a traditional and expensive clustering solution

  21. Data Warehouse Replication Sybase Business Intelligence Architecture & Ecosystem Transactions ETL Mainframe Trading Reporting / Dashboarding IBM DB2 Order Processing Quality / Audit Power-Designer IBM DataStage IBM DataQuality Microsoft SQL Server Provisioning Modeling Inventory Real-Time Report Server Oracle PowerDesigner Financials Sybase ASE

  22. PowerDesigner supports EA…..

  23. Moving from Oracle to Sybase IQ

  24. TRANSTATS • Business Issues • Congressional legislation required consolidated, single-point of access to all transportation statistics • Needed to deliver over the Web • Over 250+ databases of source data • Results • 2.5TB of detailed input data compressed to 1 TB • Query complexity with 18-way joins • Reduced data gathering time • Easy linkages across many data sets allows new insights on transportation safety • The new website is aimed at transportation researchers and analysts • Website gets 15,000 hits per day www.transtats.bts.gov Sybase IQ reduced loading and indexing from 30 minutes to 2.5 to 3 minutes. Query speeds were 20 – 50 times faster than Oracle. Time to add a column was reduced from 4 hours with Oracle to 15 minutes with IQ. Jeff Butler Assistant Director, Office of Statistical Computing Department of Transportation Bureau of Transportation Statistics

  25. YAPI KREDI BANK:Sybase IQ Blows Oracle & IBM Away • Business Challenge: • Manage and analyze immense amounts of data produced in their daily operations • Develop a smooth, fast migration path • Deploy before new credit card products and services were offered and advertised • Results: • 8 weeks to deployment • 154% return on investment • 63% data compression • Used 3 CPUs, compared to Oracle’s 12 • Queries that used to take a day are now completed in minutes “When the selection was down to two products, IBM’s UDB and Sybase IQ, we conducted tests on both. We selected Sybase IQ because it provided faster query results, reduced data storage requirements, and gave lab test performance results that were sometimes 10 times as good at the competitor.” Alpaslan OzluGeneral Manager of Technology ManagementYapi Kredi Bank

  26. TELSTRACLEAR:Sybase IQ, not Oracle, Delivers Results • Business Issues • Huge data volume – 3 billion rows of information, growing by 6 million rows per day • Goal – Develop a reporting system across the enterprise to provide on-demand reporting and business analytics against enough historical data to reflect historical trends • Results • Not able to solve need with Oracle site license • Call reporting system stored and managed 13 months of data – at no additional storage cost • Complex report times reduced to two minutes • Business analysts able to create and request their own reports – without the need for IT intervention • Increased capacity without additional hardware • Low maintenance and DBA costs “Everybody was pretty amazed. We were hoping for a 50% decrease in query time. But we saw many times more than that when we started up the system. The Sybase IQ implementation far exceeded our expectations. The results were stunning.” Simon Falconer DBA TelstraClear

  27. Oracle Complex Query Times vs IQ in Recent Customer Benchmark Hardware: IBM Power6 p570 8 cores

  28. Query Results – Financial Customer Hardware: IBM 570 systemP 16 cores(8 Dual Core)

  29. Disk Space Reduction in Customer Benchmark IQ Reduced Disk Space from 766 GB in Oracle to 240 GB

  30. Sybase IQ and Replication Server

  31. Commonwealth of PennsylvaniaDepartment of Revenue Business Issues • Reports against Mainframe data were taking too long: • Scheduled nightly • As long as a one week turn around time • Reports were extremely limited in flexibility: • Only certain columns where allowed for searching and aggregation • No true ad-hoc capabilities to drill down into details Solution • Run reports and ad-hoc queries against Sybase IQ database • Move the data to Sybase IQ • Keep data synchronized using Sybase Data Integration Suite (Replication Agent and Replication Server components) Results • Over $100 million in newly identified revenue

  32. Sybase IQ Finance Apps Commonwealth of PennsylvaniaDepartment of Revenue • Report Server • Compliance Reporting • Operational BI Existing Financial System New Financial Analytics Server Sybase DB2 Rep Agent Sybase Replication Server Staging DB Near Real-Time Updates IBM DB2 Windows Server IBM Mainframe

  33. TELEFONICA: Real-Time Analytics • Business Issues • Access to 11 legacy systems was too complex and not meeting business needs • Unable to keep up with user information demands • Needed Real-Time Information • Results • Dynamic Operational Data Store – single data storage system for all Telefónica’s operational systems – 11 operational systems, 2 billion files • Utilizes Sun-Sybase Reference Architecture, Sybase IQ and Sybase Replication Server • Implementation Time – 4 weeks • Real Time – All information updated every 5 min. since the last reading • Response time – 200 times faster than other systems tested • Cost Savings – 70 Terabytes of input information into 15 Terabytes of storage – Less hardware, support, admin & maintenance costs www.telefonica.es “The model saved between four and six times the storage volume required compared to other RDBMSs available on the market, with the attendant reduction in hardware, support, administration and maintenance costs.” Pedro Romera, Systems Engineering Manager,Telefónica

  34. Report Server Sybase IQ for Report Server

  35. Neos Banca: Financial services for millions of customers Background / Issues System serving more than four million customers across Italy Broad range of financial products In 2006, generated three billion euros worth of consumer loans Needed to optimize analytic performance without impacting transactional performance Results of the Sybase IQ Implementation Improves query response time Avoids significant data storage costs Integrates with existing technology assets easily Enhances ability to comply with Basel II and other regulatory standards Enables production of existing monthly reports on a weekly or daily basis Sybase IQ Report Server Accelerator “Sybase IQ is a unique tool. It was designed from the ground up to support analytics and its query response time is extremely fast…Also, Sybase IQ compresses raw input data up to 70 percent compared to conventional transactional-based data warehouse or reporting systems. This frees up valuable amounts of disk space and eliminates the need to invest in additional storage capacity. Without Sybase IQ, we would have had to spend a great deal of money to purchase new storage hardware.” Paolo Simonini Sr. Manager, New Technologies Neos Banca

  36. Pick-n-Pay:Reporting solution for a leading grocery retailer Issues Increased data volumes and concurrent user query and reporting demands 32-processor Teradata warehouse could not provide acceptable performance at an acceptable cost Needed a highly cost-effective, scalable reporting solution that could integrate data from many operational systems Results of the Sybase IQ Implementation Reduces annual costs Improves efficiency and access to live operational data Enhances performance and query response times Integrates easily with SAP ERP System Sybase IQ Report Server Accelerator “Even in the early stages of development, IQ is already delivering improved performance in corporate reporting.” Harold Dawson, IT Director Pick ’n Pay

  37. Accelerating Reporting Systems Offload scheduled production reports, on-demand reporting, KPIs, and dashboards from transactional systems or databases to a Sybase IQ column-based analytics server Sybase IQ Report Server Accelerator Data Warehouse's) ORDER ENTRY Sybase ASE Replication Sybase IQ BILLING DB2 INVENTORY Oracle ETL Reporting Applications and Users Operational System's) OffloadReportingData Reporting and Decision Support Systems • Designed & optimized for fast query processing • More efficient utilization of hardware resources

  38. Reporting Feels the Pinch In a data-fueled environment, reporting challenges arise: Reports take too long to run and can’t be completed during available reporting windows. Impact: missed SLAs or penalties for non-compliance by regulatory agencies Reports running against the same database used to process day-to-day business transactions cause unacceptable slowdowns in those systems. Impact: lost revenues, decline in productivity, lower customer satisfaction Current IT staff and resources are unable to support growing data volumes and reporting demands. Impact: constraints on business agility or strategic objectives Sybase IQ Report Server Accelerator

More Related