Performance Measurement in Capital Planning and Investment Control

1. Performance Measurement inCapital Planning and Investment Control Presented to Federal CPIC Forum Washington, DC July 2008

2. There is lots of information out there! “Performance Measurement” – 3,290,000 Google results “Performance Management” – 15,600,000 Google results

3. Measure what? Process input, output, outcomes Alignment Efficiency Effectiveness Quality Capability Responsiveness Risk Technical Performance Progress towards goal Satisfaction Measure how? Point in time Trend Variance (cost, schedule, other) Performance Index Maturity Model status There are lots of ways to measure performance!

4. What measures show you line-of-sight from selection criteria to business impact? $$$ per strategic goal Business impact Customer Service and Satisfaction Goals Met Business Process Improvement Technical Performance Scoring criteria IT portfolio selection criteria

5. What to talk about after breakfast???? Criteria:KISS Don’t make it too scary Make it a little challenging Therefore:Three rules for CPIC PM: 1. Address the right questions 2. Think outside the PM process 3. Don’t screw up the math!

6. What are the right questions? • Which Measures questions: • Which measures do we need? • What does this measure do for us? Is that important? Why do we care? • What is the business question this measure answers? Is that the right question? If not, what is the right question? • How is this measure defined? • How was this measure created?

7. Other Measurement questions… • How manymeasures …gauges were in the cockpit of the last airliner you boarded?

8. Other Measurement questions… How often…should we measure? • How much is at stake? • What is the response time? • What is the expected rate of change in results?

9. Thinking outside the measurement process… • Besides just reportingperformance, how well do or how much do your measures contribute to: • Improving performance? • Aligning efforts? • Managing accountability? • Streamlining work? • …and how do you know? Be able to measure the performance of your performance measurement process!!

10. Thinking outside the measurement process… • If you have a need to pay contractors and a need to collect earned value performance measures… • Make the earned value report the invoice. • If you need to monitor both risk and earned value… • check the causal analysis factors of earned value variances against what is covered in your Risk Management Plan.

11. Don’t screw up the math! (the scary part) • The bad news is generating meaning from performance data is… “statistics” • The good news is this is now junior high school math!

12. Math Basics: What’s in a Number?

13. Math Basics: Levels of Measurement Interval Number Ratio Number 1st 2nd 3rd Nominal Number Ordinal Number

14. Performance measurement math • There are sources of random error in measurement (eleven classes of error in telescopic observations in astronomy) • Variation in measurement is an inevitable by-product and not evidence of failure of the process • There are other types of measurement laws and errors. • The medical and legal professions are notorious for blowing the math – hence the “prosecutor’s fallacy” (mistake of inversion). Let’s avoid generating a “CPIC fallacy”!

15. Performance measurement math • Which system described below is more secure? Beware the law of averages Beware of composite scorings Beware of denominator management

16. Can you drown in a river that is on average only 3 feet deep? Source: Prof Sam Savage, Stanford University

17. Can you survive the Flaw of Averages? Source: Prof Sam Savage, Stanford University

18. CPIC averages… • Can a project that on average has a negative cost variance of -10% be considered a successful performer? • Can a project that has an average cost varianceof -10% be considered a successful performer? • Can a process that on average meets target productivity measures be considered successfully performing?

19. Don’t screw up the math! • Define what performance should be measured. • Learn the science of measurement: define what is relevant to be counted or measured, how we will count or measure it it, and how we will display results (Edward Tufte is your best friend here). • Learn about statistics (meaning from measures): what are the rules; what are the traps; which are required for decision-making; which are the by products of decision-making; mean, standard deviation, symmetry, kurtosis; randomness; • Realize the limits of measurement “Not everything that counts can be counted; not everything that can be counted, counts”Albert Einstein

20. Boundaries of the CPIC Universe Security

21. IT Portfolio Systems Initiative Inf Initiative Non-Sys Initiative * TBD * TBD * TBD * TBD * TBD * TBD * TBD * TBD * TBD * TBD * TBD * TBD * TBD * TBD * TBD Z Z CPIC is the Glue that holds the entire IT Universe together. APP - Outcome ---Output ------Indicators Enterprise Architecture Z * Z Z * * Z Z LEGEND Performances Measures * Initiatives Business Areas Applications Business Functions Z Business Processes Applications Technology Technology Existing Linkages Z Missing Linkages

22. Summary Bill Taylor Chief, IT Strategic Programs Division US Citizenship and Immigration Services William.R.Taylor@dhs.gov