Loading in 2 Seconds...
Loading in 2 Seconds...
Testing and go-live best practices for a successful dashboard rollout. Dr. Berg Comerit Inc. Background Planning for the Xcelsius Dashboard Initiative The JAD, RAD, Agile (XP) or ASAP methodologies Getting the Right Dashboard Requirements Upgrade and Tool Migrations
Where are my users located?
What is the network capacity?
Do I need support people for extended time periods - different time zones?
Do I need multi-currency support on my dashboards?
Do I need multi-language support?
What type of users do I have in each region?Go-Live Planning for the Xcelsius Dashboard Initiative
Create a user map as part of your project planning. This will help you understand your user base
Best when users are similar and centrally located
Best when users are dispersed, dashboards are simple or go-live over long time period
Train the trainer
Best when users resides in many locations, multiple languages are involved and when there is a very high number of users
Best for executives and senior management. Should be
done at each user's office.User Training Options
Source: dashbaord insights, 2009
Communicate and schedule training early in the project, so that everyone will be available
However, there are times when training are needed this include:
Interactive dashboards and dashboards with complex graphingUser Training for Complex Dashbaords
In this dashboard, users can budget for travel categories, for each month, and also save scenarios.
Some training is therefore required and should be planned early.
The on-line help system should explain
Create a simple help dashboard & embed this into the overall dashboard
Create a word document with screenshots.
Save it as HTML and store the web pages on a web server.
You can then link the URL on your dashboard.
Use a tool like front-page or any web authoring tool and create a complex on-line help system with menus, search functionality, movies that shows demonstrations etc.How to Create On-Line Help Systems
On-line help centers can include contact information, training schedules. They can also be used to communicate information on future projects
Joint Application Design (JAD)
Rapid Application Development (RAD)
Agile, or Extreme Programming (XP)
Accelerated SAP Methodology/ System Development Life-Cycle (SLDC)
Many of the methodologies are not appropriate for the dashboard development effortThe JAD, RAD, Agile (XP) or ASAP methodologies
Pick your Xcelsius methodology carefully.
Do not use ASAP unless you project is part of a budgeting, consolidation or planning effort.
They give a false sense of clear cut stages and does not address substantial functionality changes during development. It is hard to fix missing functionality during integration test.The "Waterfall Methodologies" are Not Good for Dashboards
The challenge with ASAP is that users don't know what they want until they see it...
2. End users – 3 to 5, attend all sessions
3. Developers – 2 or 3, question for clarity
4. Tie Breaker – Senior manager. Breaks end user ties, don’t attend
5. Observers – 2 or 3, do not speak
6. SMEs – A few subject matter experts (SME) for understanding
business & technology
Keep it very focused and explore the interfaces. How do the user want to see the screen layouts and functionality?Joint Application Design (JAD) - Who participates?
A study of 60 development projects and found that without JAD, 35% of the functionality was missed
(Source: Caper Jones)
The first meeting: a one or two days work session with
Who: Power users, casual users, people who today interact with the current
system and managers who have a stake in the outcome of the dashboards
How many: A rapid pace is kept in these meetings and the number of attendees is
kept at no more than 7 people in attendance.
The coordinators should focus on shared information needs and conduct multiple sessions (typically once a week)Rapid Application Development - RAD
Why RAD?.. Increase involvement, less business disruption, less opinions, more consensus, information sharing and an education event.
developed to build software for low levels of change and reasonably predictable outcomes.
But, the business world is no longer very predictable, and software requirements change at extremely high rates.
Development can be completed faster with collaborative efforts of
paired programmers with small 'sprint' timelines and many go-lives'.Agile and Extreme Programming (XP) for Xcelsius Dashboards
XP was started by programmers who decided that the traditional requirements gathering sessions took too much time and often just verified what they already knew.
The core premise of XP is that you can only pick 3 out of these 4 dimensions: cost, quality, scope, time.
Source: Dr. Berg,
DM Review, 2006
Discovery and Education,
Prototypes and Reviews
Final approvals.The Xcelsius Business Requirements
A dashboard implementation does not simply involve a series of black-and-white technical decisions; just because something is technically feasible does not mean it is wise or desirable from a business perspective.
Source: Gooy_GUI, 2007
This can be done in 30-60 minutes
Plan for multiple weekly prototypes before you get the solution that everyone can agree on.
Conduct a developer training or workshop session to learn the new functionality and agree on the upgrade timeline
Do a technical upgrade over the weekend
Copy all of the dashboards
Implement new functionality on the copied dashboards
Have a formal feedback session with user involvement to see how they like the changes
Implement the new changes 2-6 weeks after the technical upgradeTool Upgrades and Deployment of New Xcelsius Functionality
Stability is the key to success in all end user systems. Even if the new functionality is better, users do not like a system that changes look and feel frequently.
Maintaining two systems (not recommended)
Running two systems in parallel for short time to reconcile results
Remove legacy reporting tool as part of go-live (recommended)
When Vikings settled new lands, they always burned the boats so that the settlers was 100% committed to the new situation. While it caused some anxiety, this is a great migration strategy that can be used for system migration efforts as well.Tool Migration Strategy
A "burn the boats" reporting migration strategy assures high commitment to the new solution and possibly a high number of new requirements after go-live. Be prepared to support this...
The test plan is more detailed and should be written once the first prototype is built
During each UAT test cycle, you should solicit detailed feedback on layout, graphs, theme, tables and navigation.The Xcelsius User Acceptance Testing (UAT) Form
This form is also on your memory stick
While all areas cannot be tested, this will give you an idea on bottlenecks.
For large scale go-lives, you should consider having test PCs in multiple locationsThe Xcelsius Load Testing Form
This form is also on your memory stick
The difference is that the number of concurrent users are expected to be doubled
Not all companies will use a stress test, nor pass it. This is due to the unrealistic concurrent load and the high hardware costs of meeting them.
However, it provides very useful informationThe Xcelsius Stress Testing Form
This form is also on your memory stick
Provide food and snacks
At least two testers (preferably more) should be assigned to test each functionalityLarge-Scale Xcelsisus Test Scheduling: Example
All test results should be logged so that fixes do not impact other dashboards and consistency is maintained
The simplest way is:
Bring all content into the production box,
Setup the roles in production environment
Gradually release roles to the end users.
The benefit of doing this is that you can see how the system performs and solicit feedback from an increasingly large user group, before you release the dashboards to many users.
A gradual go-live can reduce the potentially negative impact on the organization.Big-Bang Vs. Gradual Go-lives
Unless the dashboards are for a single department, or very few users, you should always plan for a gradual go-live
Training is needed
Multi-language support is required
A high number of users are expected
Dashboards are tailored to local needs
When you use a regional go-live strategy, you can also roll out the dashboards to power users first (i.e. one month before go-live) to see how the system works in a real production settingGradual Go-lives by Region
Global dashboards require serious attention to language support, user support, 24/7 availability and attention to non-native English speakers.
Organization is very large
Departments have very different needs
Dashboards are tailored to dept. needs
Data must be secured between organizations
When you use an organizational go-live strategy, you need to focus on the branding of each dashboard (i.e. logos of subsidiaries) and on integration of support functions in their organizations (i.e. helpdesk).Xcelsius Dashboard Go-lives by Organization Units
Departmental dashboards should have first level support in their respective business units.
If there is no support organization, the BI system quickly becomes an orphan when the project ends
support org. there
is a risk that
future BI projects
are delayed since
the project team
has to support
projectsBI Support Organization — Big Picture
Note: Job areas are meant for illustrations, and will vary depending on the BI applications supported
Data loads & fixes
SAP BI Basis
i.e. WebI, BEx
Data loads & fixes
Data quality &
data resource mgmt.
When must data stores be loaded by (time)
What will happened if a persistent problem occurs (“swat” teams)?
Who is responsible for fixing process chains and who pays?
Do you get a discount for each DataStore that is not loaded in time?
How should Service Packs and Fix Packs be applied
When will SP, FP and SAP Notes be applied?
Who pays for it?
Who is responsible for testing them?
When will the BOBJ system be upgraded
When will upgrades occur, how is the pricing determined?
Who pays for it and who is responsible for testing?
How long can the system be off-line?
Minimum uptime and target uptime
What is uptime defined as (data store loaded vs. queries available vs. security fixes applied vs. portal uptime vs. third-party reporting tool uptime vs. network uptime vs. Xcelsius issues, etc.)?
What are the penalties (money) for missing the dashboard uptime requirements?
What issues must be logged?
Who owns the log? Do you have access?
Can entries be updated, or must an audit trail be preserved?
Backup and disaster recovery
What is included in the backup and when is it taken?
When will restore abilities be tested?
How fast must restore occur, and what data stores and users will first have access (priority list)?
Who owns the data
If you switch vendors, who owns the data?
How will you get access to the data? Do you get full insights to all?
Who, of the vendor’s employees, gets access to your data? Can they share it with your competitor?
When will service tickets be monitored?
What are the categories and who will resolve them?
What are the resolution process and timelines?
How are customer and support satisfaction measured?What to Include in a Dashboard BI SLA (cont.)
What will happened if an issue cannot be resolved by the internal IT department/ vendor and your business SLA manager?
What are the steps needed to terminate the SLA contract and are there any payments/fault payments or budget recourse (i.e., move money from cost centers)?What to Include in a BI Dashboard SLA (cont.)
The more details you put into the dashboard SLA up- front, the easier it will be to measure and the more likely you are to have a successful relationship
90% of all dashboards run under 20 seconds
System is available 98% of the time
Data loads are available at 8am — 99% of the time
User support tickets are answered within 30 minutes (first response)
User support tickets are closed within 48 hours — 95% of the time.
System is never unavailable for more than 72 hrs — including upgrades, fix packs, service packs, and disaster recovery
Delta backups are done each 24 hrs cycle and system backups are done every weekendReasonable Xcelsius Dashboard SLA Performance
Agile Project Dashboards - Bringing value to Stakeholders and top management (What can you do for your PO Today?)by Leopoldo Simini, July 2011
Key Performance Indicators: Developing, Implementing, and Using Winning KPIs by David Parmenter, Jan 2007Resources
Have multiple meetings with the user groups
Build interactive prototypes and expect requirements changes
Plan for a formal load testing of the dashboard
Have a rollout plan and a long-term vision of how to get there
Requirements gathering is interactive and users are discovering what they want
Spend serious time planning for a support and on-going enhancement of your dashboards, or they will become useless in a very short time...7 Key Points to Take Home
How to contact me: