Chapter 13 – Forecasting Bias

1. Chapter 13 – Forecasting Bias • Biases in forecasting • Strategies for overcoming • Expert Interview

2. Teacher Training Program Les Allais School District had been performing poorly on standardized math tests. Donald Quixote, the Superintendent of Les Allais, proposed to the school board a new mathematics curriculum for the six area high schools with 12,000 students. All 100 teachers were to be trained over the summer with the launch of the new curriculum in the fall. He predicted a significant impact within 18 months. After 2 years, there had been very modest gains. Dr. Quixote blamed the slow progress on the teachers he claimed had not fully embraced the new approach.

3. Outsource to India The VP of a German software company pushed to purchase a software support services company in Bangalore, India. The plan was to reduce the cost of global technical support for its major product lineup by eliminating its British-based English language support services. After 12 months, an analysis indicated that costs of support had gone down somewhat but that they had lost a number of major clients who had difficulty getting critical problems solved in a timely fashion.

4. Denver Police Bullet Controversy • Sequence and Linkages • Scientific Data (e.g. Muzzle velocity) • Scientific Judgment (e.g. injury) • Value Judgment : tradeoffs

5. Data + Experience  Judgment • Few technical decisions are identical. • Rarely enough data to resolve the issue with certainty. • Life and Work experiences are subject to the • “Law of Small Numbers” • Experience including a collection of rare but memorable events can be misguiding. • Single number estimates and wishful thinking produce inherent biases toward the “desired” answer. • Group involvement in every aspect compounds biases.

6. Theme: Divide & Conquer Bias(ReduceBiasa Little) • Seek experts from different areas to focus only on issues within their expertise • Design • Manufacture • Finance • Marketing • Support services • Customer relationships • Human Resources

7. Theme: Natural Forecasting Biases Can be Predicted • Biases are subconscious and subject to common patterns • Understand and recognize common forecasting biases

8. Broad Categories of Forecasting Bias • Examples • Motivational and Personal • Point estimates and too narrow ranges • Errors in Probabilistic Thinking • Availability and Representative • Confirmation

9. Examples – Bad forecast • Boston’s Big Dig $2.8B  $14.8B • 70% of new manufacturing plants close with decade • Chemical plant construction – 100% overruns (RAND) • Mergers and Acquisitions – Delusional optimism • 75% not achieve goal • 50% of time stock of purchaser declines that day • Daimler Chrysler: merge cultures

10. Motivational • Define • Chapter examples • Overestimate Mass transit ridership (economic and political) • Underestimate costs or time (Boeing) • Underestimate future corporate profit  stock price • Unrealistic optimism • Challenger shuttle and tiles • Entrepreneurs • Federal Mogul – asbestos • Personal performance – above average • What is positive value of this bias?

11. Motivational – Your examples • Corporate forecasts • Personal forecasts • Unrealistic

12. Motivational - Strategies to Overcome • Accountability for forecast • Just compare results to forecast • No vested interest – Independent – Congressional Budget Office • Corporate independence?

13. Expert Assessment Avoid Motivational Bias

14. Narrative – Probabilistic Words A project team has just reviewed plans for the launch of a new product. Each member of the team was asked to state in one word or phrase what they thought the chance of success was. Please interpret and assign a probability to each of the following ten words or phrases.

15. Phrases – Interpret

16. Form: Confidence Intervals • 0.05 0.95 (Low) (High) • Alexander Hamilton’s age at death ______ ______ Volume of water in Lake Michigan (gal.) ______ ______ Population of US in 1800 census ______ ______ Number of Chapters in Book of Psalms ______ _____ Sun’s volume as a multiple of Earth ______ ______ Weight of empty Boeing 747200 (pounds) ______ ____ Year of Da Vinci’s birth ______ ______ Average Height of Giraffe at birth (cm) ______ ______ Air distance: Detroit to Buenos Aires (miles) ______ ______ Depth of deepest point of Lake Superior (feet) ______ ______

17. Form: Confidence Interval The population of Turkey in 2010 is estimated to be 74 million. 0.05 0.95 (Low) High) 2010 Population of Greece ______ ______ More than 1 billion Muslims and more than 1 billion Christians in world 0.05 0.95 (Low) (High) 2010 World’s Jewish Population ______ ______

18. Too Narrow a Range- Illusion of Control • Define • Provide Example s • Chapter – Dreamliner, Daimler Chrysler, emerging markets and quality of service • Personal experience • Insider view – High School Example • Define • Should (insider) vs. will or has in the past (Outsider)

19. Israeli Math Course – Kahneman (Nobel laureate) and Lovallo In 1976 one of us was involved in a project designed to develop a curriculum for the study of judgment and decision making under uncertainty for high schools in Israel. The project was conducted by a small team of academics and teachers. When the team had been in operation for about a year, with some significant achievements already to its credit, the discussion turned to the question of how long the project would take. To make the debate more useful, I asked everyone to indicate on a slip of paper their best estimate of the number of months that would be needed to bring the project to a well-defined stage of completion of a complete draft ready for submission to the Ministry of Education. The estimates, including my own, ranged from 18 months to 30 months. At this point I had the idea of turning to one of our members, a distinguished expert in curriculum development, asking him a question phrased about as follows: “We are surely not the only team to have tried to develop a curriculum where none existed before. Please try to recall as many such experiences as you can. Think of them as they were in a stage comparable to ours at present. How long did it take them, from that point to complete their projects?” After a long silence, something much like the following answer was given, with obvious signs of discomfort: “First, I should say that not all teams that I can think of in a comparable stage ever did complete the task. About 40% of them eventually gave up. Of the remaining, I cannot think of any that was completed in less than seven years, nor any that took more than ten.” In response to a further question, he answered: “No, I cannot think of any relevant factor that distinguished us favorably from the teams I have been thinking about. Indeed, my impression is that we are slightly below average in terms of our resources and potential.”

20. Narrow a Range- Illusion of Predictability • Define • Provide Example • Chapter - • Move production to Canada, • Foreign currencies, • Democracy in the Middle East, • First round draft picks • Housing prices until 2007– linear extrapolation • Personal experience

21. Key Challenge: Convince Expert Probabilistic Estimate with justification requires more knowledge than Single Point Estimate. Rarity: Dilbert WRONG

22. Narrow Range- Anchoring • Define • Provide Example • Chapter • Salesman and negotiations • Black Pearls • Personal experience

23. Narrow a Range- Faulty Probability reasoning • Define • Provide Example • Chapter • Multiple independent events: • Random variable – Minimum (1st failure) and Maximum • All happening – Suppliers on time • 20 suppliers each with 95% reliability ___________? • Due for a hit, play in Super Bowl (Lions) • Bayes Rule – medical example • Personal experience

24. Availability • Define • Provide Example • Chapter • Airplane crashes • Terrorist attacks • Epidemic  pandemic 1918-1919 flu • Selling safety – Onstar • Weight loss selling • Win lottery • Personal experience

25. Representativeness • Define • Provide Example • Chapter • High-end stereo purchase • Race car driver • Interview as predictor • Parole boards – behavior in prison • Personal experience

26. Confirmation & Interpretation • Define • Provide Example • Chapter • Bush interpretation of evidence of chemical weapons • Post hoc  proctor hoc fallacy : change caused the effect • Back surgery • New leader  responsible for better or worse performance • Superstition – walking under a ladder • Personal experience

27. Expert Assessment: Expert opinion draws on relevant experience

28. Expert Assessment Expert opinion – Honest Assessment • Uncertainty and forecast a range reflects and requires more knowledge than a single point forecast • Provide basis for forecast

29. General Guidelines: Expert Interview • Unwise for an individual to attempt to assess his own probabilities • Expert simply filling in a form on his own  meaningless data • Do not know assumptions expert used in recording values • Do not know motivations at play when form was filled out • Do not even know if request was taken seriously. • Meaningless examples ________________________ • Try to understand Expert’s cues and modes of information processing to infer biases likely to exist in responses

30. General Guidelines: Expert Interview • Encourage expert to clarify assumptions • Challenge expert to justify - non-threatening manner • Direct expert to draw on “appropriate experiences” • Interview will take from 1 to 2 hours. • Note taker in addition to interviewer enables interviewer to focus and hear the nuances of expert

31. Expert Interview • Prepare to Interview Learn enough to design intelligent interview • Motivate IntervieweeEstablish rapport and explore personal biases • Structure Interview Exact specification of uncertain variables • Condition Interviewee Describe process for exploring ranges and potential cognitive biases. • Elicit and Encode Data Elicit probabilistic ranges for key variables and convert answers to probability distribution • Verify Answers Review answers and results to ensure final distribution reflects expert’s assessment of uncertainty • Aggregate Information Bring together all uncertainties

32. Prepare for InterviewDesign Intelligent Interview • Divide the decision context into distinct areas requiring unique expertise (e.g. manufacturing, finance, R&D) • Identify recognized and accepted subject matter expert(s) • Clarify their area of expertise • Scope out the focus of the interview: gather information but not specific numeric values • Plan out initial questions and strategies for known biases • Biases – Expert’s Background • Well known commitment to specific technologies • Corporate sponsor of “technology x” • Recent major successes or failures

33. Preparation for Expert Interview: Survey FormExample from R&D and Process improvement Projects in Glass Division • Describe in two paragraphs the overall nature of the research project • Discuss Current specific focus (or foci) of research • What is the current capability of other car companies, glass companies or other organizations with regard to this project and potential products? • What opportunities exist for technical collaboration? • What is the next technical hurdle(s) to overcome before moving on to a subsequent phase? What other major technical hurdles lie ahead? • What would the end product in the car be? or the end product in terms of a Process Improvement? • What are some key performance characteristics of the proposed product or improvement?

34. Preparation for Expert Interview: Form continuedExample from R&D and Process improvement Projects in Glass Division • What government regulations (current and proposed) could impact the minimum standards for this product or process improvement? • Describe the potential need for fleet testing. • What MAJOR changes in manufacturing processes or investments might be needed to support the production of the related new product or introduction of the process improvement? • The integration of new products into a car has different lead times. Privacy glass has a short lead-time and specially shaped windshields have long lead times. What is the latest stage in the car design process during which this product could still be incorporated into a new model? • The actual form is two pages long and has enough room for responses to each question.

35. In-Class Interview • Teams of 3 or 4 • Pick an uncertain variable in your work environment • Think about an interview • Carry out interview

36. Interview Summary Form  Technical and business uncertainty plus strategic value (multiple objectives)

37. Three-point Approximation

38. Partial Explanation

39. Translate into Tree Probabilities