Delivering strategic insight
Download
1 / 56

OCBIG PRESENTATION BI Tool Evaluations A Methodology and Case Studies - PowerPoint PPT Presentation


  • 113 Views
  • Uploaded on

“Delivering Strategic Insight”. OCBIG PRESENTATION BI Tool Evaluations A Methodology and Case Studies. Robert Snoyer Managing Partner Steve Jones Managing Partner. April 28, 2005. Agenda. Red Sky Overview An Approach to BI Tool Evaluations Components Methodology Case Study #1 – Telco

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'OCBIG PRESENTATION BI Tool Evaluations A Methodology and Case Studies' - teresa


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Delivering strategic insight l.jpg

“Delivering Strategic Insight”

OCBIG PRESENTATIONBI Tool EvaluationsA Methodology and Case Studies

Robert SnoyerManaging PartnerSteve JonesManaging Partner

April 28, 2005


Agenda l.jpg
Agenda

  • Red Sky Overview

  • An Approach to BI Tool Evaluations

    • Components

    • Methodology

  • Case Study #1 – Telco

  • Case Study #2 – Health Care government agency

  • Questions

BI Tool Evaluations: A Methodology & Cases Studies


Agenda3 l.jpg
Agenda

  • Red Sky Overview

  • An Approach to BI Tool Evaluations

    • Components

    • Methodology

  • Case Study #1 – Telco

  • Case Study #2 – Health Care government agency

  • Questions

BI Tool Evaluations: A Methodology & Cases Studies


Red sky data provides solutions l.jpg
Red Sky Data Provides Solutions

Red Sky Datais a Canadian consulting company that specializes in providing:

  • Business Performance Management (BPM)

  • Business Intelligence (BI)

  • and Data Warehouse (DW) solutions

    The partners of Red Sky Data have provided cutting edge solutions for several decades

BI Tool Evaluations: A Methodology & Cases Studies


Red sky s lineage l.jpg

Partners leave to start DVI-Dec ’98

RSI1989 – Mar ’00

Data VisionsDec ’98 – Jun ’01

CGI

Red Sky DataJan ’05 – present

CognicaseJun ’01 – Jan ’03

Microforum

Acquired by Cognicase-Jun ’01

Acquired by CGI-Jan ’03

Acquired byMicroforum-Mar ’00

Partners form RSD -Jan ’05

consulting group acquired by Cognicase-Sept ’02

Red Sky’s Lineage

BI Tool Evaluations: A Methodology & Cases Studies


Key differentiator the team l.jpg
Key Differentiator -The Team -

  • 22 data warehouse and business intelligence specialized consultants

  • Industry-recognized technical experts

  • Regular presenters at industry seminars

    • TOUG, IOUG, Informatica World, Oracle World, Master Classes, IRMAC, OCBIG

  • Certified professionals

    • Informatica, Oracle, MicroStrategy, Cognos, Microsoft

  • Dedicated to the client, our organization, and each other

BI Tool Evaluations: A Methodology & Cases Studies


Key differentiator advantagenow v4 0 l.jpg
Key Differentiator - AdvantageNOW v4.0 -

  • AdvantageNOW is a framework for delivering BPM / Data Warehouse and Business Intelligence projects on time and on budget

  • Based on Ralph Kimball and “best practices”

  • Continuous improvement through real-life team experiences – living framework

  • Comprised of:

    • Methods

      • Loading

      • Cleansing

      • Data Retention and Archiving

      • Reporting

    • Templates

    • Project Management Processes (PMI)

    • Industry specific designs and code examples

BI Tool Evaluations: A Methodology & Cases Studies


Red sky offers full bi dw solutions l.jpg

Solutions

Enterprise

BI

Technology

  • Data Warehouse

    (DW)

  • Data Marts (DM)

  • Extract, Transform,

    Load (ETL)

  • BI Front-End

  • Meta Data

Bus. Focused

OLAP

Ad-Hoc Query

Reporting

BPM

Dashboard (KPI’s)

Visualization

  • Data Marts (DM)

  • Extract, Transform,

    Load (ETL)

  • BI Front-End

  • Meta Data

ETL

Informatica

DataStage

DTS

Services

BI

Strategic Services

Cognos

MicroStrategy

Business Objects

Readiness Assessments

Product Evaluations

Audits

Proof of Concepts

Strategies

Scope Exercises

Training & Education Services

Database

Mentoring & Training

Best Practices & Approaches Workshops

Oracle

DB2

SQLServer

Red Sky Offers Full BI/DW Solutions

AdvantageNOW Framework

BI Tool Evaluations: A Methodology & Cases Studies


Our bi dw expertise by vertical l.jpg
Our BI/DW Expertise By Vertical

Government

  • Cancer Care Ontario

  • Greater Toronto Airport Authority (GTAA)

Fin. Services

  • Dynamic Mutual Funds

  • ICICI Bank of Canada

Communications

Allstream

Rogers Cable Systems

Rogers Wireless

Telus Mobility

Carswell Thompson

MRD

  • McDonalds

  • Toyota

  • Direct Energy

  • Cardinal Meat Specialists

BI Tool Evaluations: A Methodology & Cases Studies


Agenda10 l.jpg
Agenda

  • Red Sky Overview

  • An Approach to BI Tool Evaluations

    • Components

    • Methodology

  • Case Study #1 – Telco

  • Case Study #2 – Health Care government agency

  • Questions

BI Tool Evaluations: A Methodology & Cases Studies


Components of an evaluation l.jpg
Components of an Evaluation

  • Short-Listing

  • Paper Evaluation

  • RFP

  • Vendor Presentations

  • Proof of Concept

  • Bake-Off

BI Tool Evaluations: A Methodology & Cases Studies


Short listing steps duration l.jpg
Short-Listing- Steps & Duration -

  • Develop 5-10 broad criteria for selection (ie in business >10 years)

  • High level research of the tools on the market for the purposes of answering your evaluation criteria

  • Prepare your findings

Duration: 1 week

BI Tool Evaluations: A Methodology & Cases Studies


Short listing strengths weaknesses l.jpg

STRENGTHS

Greatly simplifies all future steps

WEAKNESSES

Possibility of prematurely eliminating a valid contender

Short-Listing- Strengths & Weaknesses -

BI Tool Evaluations: A Methodology & Cases Studies


Paper evaluation steps duration l.jpg
Paper Evaluation- Steps & Duration -

  • Gain a good understanding of this market segment and what makes for a successful BI tool (see Sources later)

  • Gather the business requirements from your users

  • Develop the Evaluation Criteria

  • Conduct in-depth research into the short-listed tools and their capabilities (see Sources later)

  • Evaluate the tools against your criteria

  • Prepare your findings

    Duration: 2-3 weeks

BI Tool Evaluations: A Methodology & Cases Studies


Paper evaluation sources l.jpg
Paper Evaluation- Sources -

  • Organizations / Web-Sites

    • The Data Warehouse Institute (TDWI) tdwi.org

    • olapreport.com

    • datawarehouse.com

    • Datawarehousing.com

    • datawarehousingonline.com

  • Industry Publications

    • Intelligent Enterprise (intelligenctenterprise.com)

    • DM Review (dmreview.com)

  • Vendor Web-Sites

BI Tool Evaluations: A Methodology & Cases Studies


Paper evaluation sources16 l.jpg
Paper Evaluation- Sources -

  • IT Advisory Groups

    • Gartner Group

    • Giga Information Group

    • Yankee Group

    • Meta Group

  • Body of Knowledge

    • Participation in previous evaluations

    • Experience with the tools

BI Tool Evaluations: A Methodology & Cases Studies


Paper evaluation strengths weaknesses l.jpg

STRENGTHS

Quickest turn-around time

Least demand on resources ($ and people)

WEAKNESSES

Doesn’t provide opportunity to see the tools in action (look & feel)

Doesn’t provide opportunity to clarify information with the vendors

Doesn’t validate the features and functionality

Paper Evaluation- Strengths & Weaknesses -

BI Tool Evaluations: A Methodology & Cases Studies


Rfp steps duration l.jpg
RFP- Steps & Duration -

  • Formalized version of the Paper Evaluation

  • Same steps as the Paper Evaluation, with the addition of:

    • Prepare and distribute the RFP document

    • Provide for a formal Q & A process

      Duration: 4-8 weeks

BI Tool Evaluations: A Methodology & Cases Studies


Rfp strengths weaknesses l.jpg

STRENGTHS

Places the onus on the vendor to provide the responses

Seen as more impartial and legally binding

Opportunity to clarify information with the vendor

WEAKNESSES

Doesn’t provide opportunity to see the tools in action

Doesn’t validate the features and functionality (but are they legally bound anyway?)

Additional effort to prepare the RFP (beyond preparing the criteria)

RFP- Strengths & Weaknesses -

BI Tool Evaluations: A Methodology & Cases Studies


Vendor presentations l.jpg
Vendor Presentations

  • Must be focused - ‘Dog & Pony’ shows not permitted

  • How do you guarantee this?

    • Detailed evaluation criteria

    • Involve both business and technical teams

    • Clearly communicate the format, criteria and purpose of the presentations to the vendors

BI Tool Evaluations: A Methodology & Cases Studies


Vendor presentation steps duration l.jpg
Vendor Presentation- Steps & Duration -

  • Prepare the detailed evaluation criteria

  • 1st presentation: focused on business users and business functionality (‘day in the life of a user’)

  • Post-presentation review and scoring

  • 2nd presentation: focused on IT users and technical functionality

  • Post-presentation review and scoring

  • Summarize and prepare findings

Duration: 4 weeks + (depending on # of vendors)

BI Tool Evaluations: A Methodology & Cases Studies


Vendor presentations strengths weaknesses l.jpg

STRENGTHS

Opportunity to clarify information with the vendor

Opportunity to see the tools in action

WEAKNESSES

Doesn’t validate the features and functionality

As you increase the # of vendors it can become confusing (‘they meld into one’)

Increasing demand on resources ($ and people)

Vendor Presentations- Strengths & Weaknesses -

BI Tool Evaluations: A Methodology & Cases Studies


Proof of concept l.jpg
Proof of Concept

  • Single vendor brought in to prove their tool meets your functional requirements

  • Typically done in conjunction with a Paper Evaluation or an RFP

BI Tool Evaluations: A Methodology & Cases Studies


Proof of concept steps duration l.jpg
Proof Of Concept- Steps & Duration -

  • Select the vendor for the PoC (Paper Eval, RFP, other)

  • Develop the criteria and use cases for the PoC

  • Setup the environment (servers, s/w)

  • Determine the subject area for PoC

  • Find/build data model, data sources and business rules

  • Develop the BI environment (cubes, reports)

  • Put the tool through its paces

  • Evaluate the outcome against expected results

  • Prepare your findings

Duration: 2-4 weeks (not including effort for item 1)

BI Tool Evaluations: A Methodology & Cases Studies


Proof of concept strengths weaknesses l.jpg

STRENGTHS

Opportunity to see the tool in action

Opportunity to clarify information with the vendor

Validates the features and functionality

WEAKNESSES

Doesn’t provide a ‘safety valve’ should the tool not meet your expectations

Increasing demand on resources (if it is one of the components of your BI Evaluation process)

Proof of Concept- Strengths & Weaknesses -

BI Tool Evaluations: A Methodology & Cases Studies


Bake off l.jpg
Bake-Off

  • Just an extension of the PoC

  • Involves 2-3 vendors putting their tool through the paces

  • Provides a ‘safety valve’ should one (or two) tools not meet your expectations

BI Tool Evaluations: A Methodology & Cases Studies


Bake off steps duration l.jpg
Bake-Off- Steps & Duration -

  • Same process as the PoC, with the addition of these steps for the 2nd/3rd tool:

    • Develop the BI environment (cubes, reports)

    • Put the tool through its paces

    • Evaluate the outcome against expected results

      Duration: 5-8 weeks (not including effort to select the short-listed tools)

BI Tool Evaluations: A Methodology & Cases Studies


Bake off strengths weaknesses l.jpg

STRENGHTS

Opportunity to see the tool in action

Opportunity to clarify information with the vendor

validates the features and functionality

Provides a ‘safety valve’

WEAKNESSES

Greatest demand on resources ($ and people)

Bake-Off- Strengths & Weaknesses -

BI Tool Evaluations: A Methodology & Cases Studies


Red sky s evaluation criteria methodology l.jpg
Red Sky’s Evaluation Criteria Methodology

  • Process to Create Evaluation Criteria

    • Determine the evaluation criteria (70-90)

    • Determine the ‘key differentiators’ – Type II

    • Other criteria are the ‘base criteria’ – Type I

    • Group the criteria into categories

  • Evaluation Criteria Scoring

    • Type II Criteria – 5 points (0-5)

    • Type I Criteria – 1 point (0-1)

    • Prioritize the top 4 categories and weight them 5-2. All other categories are weighted 1

    • Final category scores based on: (Avg. Type I score x 1) + (Avg. Type II score x 5)

BI Tool Evaluations: A Methodology & Cases Studies


Red sky s poc bake off methodology l.jpg
Red Sky’s PoC/Bake-Off Methodology

  • Preparation

    • Identify ‘key differentiators’, ‘base cases’ & categories

    • Create 3-5 use cases (i.e. build a specific report)

    • Create an expected results ‘answer sheet’

    • Involve the users throughout the process

  • Execution

    • Assign an IT resource to work closely with the vendor throughout the build process (learning & assessment)

    • Eliminate generic BI learning curve by having users evaluate the tools in small, parallel steps

    • Lots of hands-on time for users

BI Tool Evaluations: A Methodology & Cases Studies


Agenda31 l.jpg
Agenda

  • Red Sky Overview

  • An Approach to BI Tool Evaluations

    • Components

    • Methodology

  • Case Study #1 – Telco

  • Case Study #2 – Health Care government agency

  • Questions

BI Tool Evaluations: A Methodology & Cases Studies


Approach l.jpg
Approach

  • Short-Listing

  • Vendor Presentations

  • Bake-Off

BI Tool Evaluations: A Methodology & Cases Studies


Short listing l.jpg
Short-Listing

  • Client was very familiar with BI technology and knew what was out there

  • Joint decision on the vendors was based on the clients knowledge and our experience

  • Decision was made to evaluate 2 vendors

    • Duration: < 1 week (1 week was the bench-mark)

BI Tool Evaluations: A Methodology & Cases Studies


Vendor presentations steps duration l.jpg
Vendor Presentations- Steps & Duration -

  • Prepare the detailed evaluation criteria

  • 1st presentation: focused on business users and business functionality (‘day in the life of a user’)

  • Post-presentation review and scoring

  • 2nd presentation: focused on IT users and technical functionality

  • Post-presentation review and scoring

  • Summarize and prepare findings

    Duration: 6 weeks (4 weeks was the bench-mark)

BI Tool Evaluations: A Methodology & Cases Studies


Vendor presentations vendor evaluation criteria l.jpg
Vendor Presentations - Vendor Evaluation Criteria -

  • Evaluation Criteria Metrics

    • Evaluation criteria (75)

    • ‘key differentiators’ – Type II (30)

    • ‘base criteria’ – Type I (45)

    • Criteria categories (12)

  • 12 Categories

    • Functionality

    • Ease of Use/Manageability

    • Training and Support

    • Meta Data

    • Security

    • Debugging/Error Handling

  • Deployment Management

  • Scheduler Capabilities

  • Change Management

  • Administration/Operational

  • Backup and Recovery

  • Technical Environment

BI Tool Evaluations: A Methodology & Cases Studies


Vendor presentations vendor evaluation criteria36 l.jpg
Vendor Presentations - Vendor Evaluation Criteria -

  • Evaluation Criteria Scoring

    • Type II Criteria – 5 points (0-5)

    • Type I Criteria – 1 point (0-1)

    • Prioritize the top 5 categories and weight them 5-2. All other categories are weighted 1

    • Final category scores based on: (Avg. Type I score x 1) + (Avg. Type II score x 5)

  • Top 5 Categories

    • 5 - Functionality

    • 4 - Ease of Use/Manageability

    • 3 - Training and Support

    • 2 - Meta Data

    • 2 - Security

BI Tool Evaluations: A Methodology & Cases Studies


Vendor presentations sample detailed evaluation l.jpg
Vendor Presentations- Sample Detailed Evaluation -

  • Sample scoring for a category with 4 base criteria and 2 key differentiators

    • Note how A and B have the same actual score but C has a higher average due to a better key differentiator score

BI Tool Evaluations: A Methodology & Cases Studies


Vendor presentations evaluation summary l.jpg
Vendor Presentations- Evaluation Summary -

BI Tool Evaluations: A Methodology & Cases Studies


Bake off steps and duration l.jpg
Bake-Off- Steps and Duration -

  • Select the vendors (paper eval, RFP, vendor presentation)

  • Develop the evaluation criteria and use cases

  • Setup the environment (servers, s/w)

  • Determine the subject area for the Bake-Off

  • Find/build data model, data sources and business rules

  • Develop the BI environment (1 cube, 3 reports)

  • Put the tool through its paces

  • Evaluate the outcome against expected results

  • Prepare your findings

  • Duration: 6 weeks (5-8 weeks was the bench-mark)

BI Tool Evaluations: A Methodology & Cases Studies


Bake off evaluation criteria l.jpg
Bake-Off- Evaluation Criteria -

  • All Type I criteria were eliminated

  • Type II’s were re-visited to identify the most critical (9 removed, leaving 21)

  • This reduced the Categories from 12 to 8

  • Remaining Categories

    • Functionality

    • Ease of Use/Manageability

    • Meta Data

    • Security

    • Deployment Management

    • Scheduler Capabilities

    • Administration/Operational

    • Technical Environment

  • Eliminated Categories

    • Change Management

    • Backup and Recovery

    • Debugging/Error Handling

    • Training and Support

BI Tool Evaluations: A Methodology & Cases Studies


Bake off evaluation criteria41 l.jpg
Bake-Off- Evaluation Criteria -

Chose 5 components and weighted them according to importance

  • 45% - Use Cases

    • Built to test the 21 Evaluation Criteria

    • Business – 3/4

    • IT – 1/4

  • 25% - Product Pricing

  • 10% - Customer Support & Service

  • 10% - Development Evaluation

  • 10% - User’s Personal Experiences

    100%

  • BI Tool Evaluations: A Methodology & Cases Studies


    Bake off sample detailed bake off evaluation l.jpg
    Bake-Off - Sample Detailed Bake-Off Evaluation -

    • The Business Use Cases with the detailed use cases for the Canned Reports Features

    BI Tool Evaluations: A Methodology & Cases Studies


    Bake off evaluation summary l.jpg
    Bake-Off- Evaluation Summary -

    116/180 x 45 = 29

    BI Tool Evaluations: A Methodology & Cases Studies


    Agenda44 l.jpg
    Agenda

    • Red Sky Overview

    • An Approach to BI Tool Evaluations

      • Components

      • Methodology

    • Case Study #1 – Telco

    • Case Study #2 – Health Care government agency

    • Questions

    BI Tool Evaluations: A Methodology & Cases Studies


    Approach45 l.jpg
    Approach

    • RFP

    • Short-Listing

    • Proof of Proposals (POP’s) – Part 1

    • Proof of Proposals (POP’s) – Part 2

    BI Tool Evaluations: A Methodology & Cases Studies


    Slide46 l.jpg
    RFP

    • Client was new to BI technology

    • Very aggressive timelines

    • Assisted client in preparing RFP and facilitating RFP evaluation process

    • Defined Mandatory and Rated Requirements

    • Critical Rated Requirements = “Key Differentiators”

      Duration: 6 weeks (4-8 weeks was the bench-mark)

    • 12 Categories

      • Functionality

      • Ease of Use/Manageability

      • Training and Support

      • Meta Data

      • Security

      • Debugging/Error Handling

    • Deployment Management

    • Scheduler Capabilities

    • Change Management

    • Administration/Operational

    • Backup and Recovery

    • Technical Environment

    BI Tool Evaluations: A Methodology & Cases Studies


    Short listing47 l.jpg
    Short-Listing

    • RFP responses evaluated against:

      • Mandatory (18)

      • Rated Requirements (140)

    • Rated Requirements:

      • Assigned a score (1 - lowest to 5 - highest)

      • Assigned a weight (1 - lowest to 5 - highest)

      • Highest possible score for any one requirement: 5x5=25

    BI Tool Evaluations: A Methodology & Cases Studies


    Rated requirements scoring sample l.jpg
    Rated Requirements Scoring- Sample -

    BI Tool Evaluations: A Methodology & Cases Studies


    Short listing con t l.jpg
    Short-Listing - Con’t

    • Financial responses evaluated=TCO over 5 years:

      • Initial price per component

      • Annual support and maintenance

      • Development, Test, Production environments

    • Best Value Determination=Overall Weighted Score

      • 80% - Rated Requirements scores

      • 20% - Price

    • Top 3 vendors selected to proceed to PoP process

      Duration: < 1 week

    BI Tool Evaluations: A Methodology & Cases Studies


    Best value determination example l.jpg
    Best Value Determination- Example -

    BI Tool Evaluations: A Methodology & Cases Studies


    Proof of proposals pops part 1 l.jpg
    Proof of Proposals (PoPs)- Part 1 -

    • Conducted onsite using client equipment

    • Vendors provided data model and “storyboard” in advance

    • Storyboard focused on Key Differentiators

    • Each vendor allocated 3 days:

      • 1 day for install

      • 1 day for prep

      • 1 day for demo to technical and business audiences

        Duration: 2 weeks

    BI Tool Evaluations: A Methodology & Cases Studies


    Proof of proposals pops part 1 con t l.jpg
    Proof of Proposals (PoPs) - Part 1 – Con’t -

    • Debriefing with assessment team immediately following PoP

    • Rated Requirements scores updated based on PoP results

    • Top 2 selected to proceed to second PoP phase

      Duration: 2 weeks

    BI Tool Evaluations: A Methodology & Cases Studies


    Proof of proposals pops part 2 l.jpg
    Proof of Proposals (PoPs) - Part 2 -

    • Extended demonstration of product capabilities

    • Vendors given opportunity to “wow” business audience

    • Each vendor allocated 5 days:

      • 1 day for install

      • 2 days for prep (working closely with business reps)

      • 1-2 days of demo and hands-on for business

    • Winning vendor selected

      Duration: 2 weeks

    BI Tool Evaluations: A Methodology & Cases Studies


    Lessons learned l.jpg
    Lessons Learned

    • Choose a business sponsor

      • Will engage the business and establish the subject area

    • Ensure you have significant involvement from IT and the business throughout the process

      • All criteria and business cases validated by the client

    • Keep it simple

      • Don’t attempt to incorporate the evaluation with a project deliverable

      • Minimal criteria

    • Be prepared for surprises

      • Vendor added at the 11th hour

      • Decision maker “suggesting” the appropriate vendor

      • Project gets cancelled

    BI Tool Evaluations: A Methodology & Cases Studies


    Questions answers and discussion l.jpg
    Questions, Answers and Discussion

    BI Tool Evaluations: A Methodology & Cases Studies


    Thank you for your time and interest l.jpg
    Thank-You for your Time and Interest

    Robert Snoyer

    Managing Partner

    Red Sky Data Inc.

    [email protected]

    416-712-7499

    Steve Jones

    Managing Partner

    Red Sky Data Inc.

    [email protected]

    416-524-9157

    www.redskydata.com

    BI Tool Evaluations: A Methodology & Cases Studies


    ad