1 / 52

Improving accuracy in inefficient firm level forecasts: with lessons for macro-forecasters.

Research Grants GR/60181/01 and GR/60198/01. Improving accuracy in inefficient firm level forecasts: with lessons for macro-forecasters. . Robert Fildes Lancaster University Centre for Forecasting & Paul Goodwin, University of Bath Supported by Kostas Nikolopoulos

sarah
Download Presentation

Improving accuracy in inefficient firm level forecasts: with lessons for macro-forecasters.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Research Grants GR/60181/01 and GR/60198/01 Improving accuracy in inefficient firm level forecasts: with lessons for macro-forecasters. Robert Fildes Lancaster University Centre for Forecasting & Paul Goodwin, University of Bath Supported by Kostas Nikolopoulos Wing Yee Lee, University of Bath An EPSRC Research Project A collaboration between Lancaster Centre for Forecasting,, and 10+ companies, including McBride, Interbrew, Heinz, Cow and Gate

  2. Outline • How and why forecasting is carried out at the company level • Similarities and differences compared to macroeconomic forecasting • Analysis of company level forecasts • Explaining the excessive use of judgmental interventions • Potential lessons for macroeconomic forecasting

  3. Focus: • on short and medium term demand forecasting in supply chain companies • Forecasts in these companies are used in decisions relating to: - logistics, - human resource planning, - stock control, - purchasing, - cash flow management…. Service - inventory investment tradeoff curves The wrong product in the wrong place at the wrong time - forecasters motivated to improve accuracy

  4. Promotion: 2 for 1 Economic factors Customers £ Sales EPOS info Retailer Orders Shipments Manufacturer Information affecting the supply chain The full model: Orders=f(past orders, Sales, forecast sales, promotions, Events) The basic model: Orders=f(past orders) + judgemental estimates of promotions

  5. A complete statistical model?Is it possible? • The Problems • Too complex • Incomplete data on many drivers • ‘Unique’ events • No available statistical expertise • Management understanding & acceptance • Belief that managerial expertise is best • Cost of cleaning past data to remove unusual events Marketing factors The Practical Solution: Capture the unusual complexity by managerial judgement

  6. Organisationally based Forecasting combines statistical analysis with managerial judgement Complementary nature of statistical forecasts & management judgment • humans are adaptable and can take into account one-off events, but they are inconsistent and suffer from cognitive biases • statistical methods are rigid, but consistent, and can take into account large volumes of information

  7. Forecast How companies make their forecasts 1 Package embodying simple robust time series methods is used(e.g. simple exponential smoothing, Holt-Winters) • Statistical forecast can then be judgmentally adjusted, usually at a meeting. – ostensibly for special events like promotion campaigns

  8. Differences between macroeconomic and company forecasting • Very large numbers of series need to be forecast regularly in companies • Macro forecasters are usually statistically trained, company forecasters are not • Unlike macro-economic forecasting there are usually no company forecasts that compete to forecast the same variable • Macro forecast variables are at a higher level of aggregation, but lower frequency • Macro forecasters employ explanatory models, company forecasters tend to use univariate methods

  9. Similarities with macroeconomic forecasting • Process of forecasting similar • Judgment used in model selection and fitting and in subsequent adjustment of resulting forecast • Difficult to tease out these separate effects of judgment • Starting conditions often not known for certain

  10. The EPSRC Research Project Company Evidence Data (4 U.K. based companies) • 578 SKUs, 3668Months • Company A: Major UK Manufacturer of Laundry, household cleaning and personal care products - 234 SKUs x 9 months -> 908 triplets • Company B: Major International Pharmaceutical Manufacturer - 134 SKUs x 24 months -> 2023 triplets • Company C: Major International canned Food Manufacturer • 210 SKUs x 6 months -> 908 triplets • 548 SKUs, 2039Weeks • Company D: Major UK Retailer (over 26000 SKUs) - 548 SKUs x 52 weeks -> 2039 triplets

  11. Research Issues and Hypotheses (practical and theoretical) • Does adjustment improve accuracy? • Under what circumstances • Are the organisational forecasts rational? • Do inefficiencies arise from organisational processes • Can the forecasts be improved? • By Model based improvements • By software improvements • By process improvements

  12. Evidence on frequency of judgmental adjustments from 4 of our companies A significant % of forecasts are judgmentally adjusted

  13. Results: MAPE by Adjustment Size Small Large Adjustments Adjustments ALL Monthly data - Final vs. System + Size Error Final Forecast System Forecast

  14. Results Company D - Final vs. System + Size

  15. For first two companies larger adjustments tend to lead to the greatest gains in accuracy Do not make small adjustments! SFC= statistical forecast

  16. The Types of Error • Adjustments in the wrong direction • With expected positive information • Adjust up, but actual is below system: Final> System> Actual • With expected negative information • Adjust down, but actual is above system • Adjustment in the right direction • Adjustment is not strong enough • With positive info, System>Final> Actual • With negative info, Actual<Final<System • Adjustment is too strong • With Positive info, System<Actual< Final • With Negative info, Final<Actual<System

  17. Adjustments made in the wrong direction reduce accuracy SFC= statistical forecast

  18. Bias Fcasts Fcasts

  19. Modelling the forecastsStatistical Issues For unbiasedness • Errors heteroscedastic with outliers • Can firms be pooled? • Solutions • Errors normalised by standard deviation of actuals and analysed by size of adjustment

  20. Bias of final forecasts Also evidence that for A,B and C companies bias is reduced through judgmental adjustment

  21. Can we model the error to ensure an efficient forecast? improved forecast error The models: Adjust is proxy for ‘Market intelligence Efficiency = all available information is being used effectively The last- the 50/50 model: Blattberg & Hoch

  22. The Effects of Information • Define Positive (negative) information as when the system forecast is adjusted upward (downward) • Questions: • Are there differences in accuracy? • Are there differences in efficiency? • yes: affecting both mean bias and regression bias • Can these be capitalised on to design rules to improve accuracy?

  23. Comparative Results: Major gains with some companies

  24. Weighting the Information Sources

  25. Conclusions from the Empirical Analysis • 1 out of 3 adjustments is in the wrong direction! • They are very costly b) Inefficiencies persist • effects are exaggerated for positive info • Double counting? Wishful thinking? c) Small adjustments (<10-20%) should not be made! d) Final forecasts are (usually) more accurate than system forecasts e) Reweighting of system forecast and market adjustment leads to improved accuracy - is it the data, the drivers or the forecasting process that makes the difference? Can we by understanding the process, develop an FSS that supports improved accuracy

  26. The adjusted forecasts are biased and inefficient: the process through which the forecasts are adjusted and the Market Intelligence estimated leads to inaccuracies … Hence there may be gains in improving the quality of judgmental inputs to forecasts But most forecasting software do not provide facilities to support judgment. Instead, packages promote their statistical power…..! • To understand the process of adjustment • To influence the adjustments through better FSS design Aim:

  27. Why do managers adjust forecasts excessively? • The ‘advice’ perspective • We can regard the computer system’s forecasts as a form of advice • Our willingness to accept advice from a machine can vary…

  28. Theories of why we accept advice Yaniv & Kleinberger (2000) : • People trust their own forecasts more because they have greater access to the rationale for these - No rationale or explanation is provided in most forecasting software packages employing univariate methods

  29. Theories of why we accept advice Yaniv & Kleinberger (2000): • Weight attached to advice is dependent on reputation of adviser - but negative information about an adviser is perceived to be more diagnostic than positive -In forecasting, noise & special events may contribute to a negative perception • Kaplan, et al (2001) found people were more likely to rely on a system when its accuracy was not disclosed.

  30. Why do managers adjust forecasts excessively? • Misconceptions of randomness We tend to see systematic patterns in what are really random movements in graphs….

  31. Would you want to adjust the statistical forecast for month 21 (below)?

  32. Ability to manipulate the system and carry out ‘what if’ analyses leads to an illusion of control (Davis et al 1994) -manipulation involves effort and is perceived to involve skill -people become overconfident in their judgments

  33. Manipulations Change the responsiveness of the forecasts… Change the forecasting method... Change the amount of past data used..

  34. Result of overconfidence Over weighting of judgmental forecasts relative to statistical forecasts -even when evidence shows judgment is less accurate…. In one study people continued to rely on their judgment despite receiving messages from the computer like… “Please be aware that you are 18.1% LESS ACCURATE than the statistical forecast provided to you.”

  35. Why do managers adjust forecasts excessively? 4. Confusion between forecasts and: -targets -decisions -politically acceptable numbers A forecast: “I think demand will be 200 units” A decision “I think we ought to produce 250 units, in case demand is unexpectedly high”

  36. Why do managers adjust forecasts excessively? 5. Need for ‘ownership’ of the forecasts • Demonstration that you’ve contributed to the forecasting process and earned your salary • Demonstration of your marketing expertise to your colleagues at meetings

  37. "Tom W. is of high intelligence, although lacking in true creativity. He has a need for order and clarity, ...... in which every detail finds its appropriate place. His writing is rather dull and mechanical, … enlivened by somewhat corny puns …… He has a strong drive for competence. He seems to feel little sympathy for other people …. Self-centered, he nonetheless has a deep moral sense.” 6. Base-rate information vs Case-specific information Kahneman and Tversky: Group A: This description has been drawn randomly from a folder containing descriptions of 30 engineers & 70 lawyers Group B: This description has been drawn randomly from a folder containing descriptions of 70 engineers & 30 lawyers -What is the probability that Tom W is an engineer?

  38. People over-emphasise case specific information at the expense of base-rate information • Our company forecasters focused on each demand figure on the graph as being a special case. -the base rate information represented by the statistical forecasts was thus under weighted

  39. Why do managers adjust forecasts excessively? 7. Recency bias - only the recent past was thought to be relevant • Statistical methods were fitted to very short sets of data (often 6 months to 2 years) • Statistical methods not given much of a chance -reinforcing the forecasters’ confidence in the relative accuracy of judgment

  40. Potential lessons for macroeconomic forecasting 1. Lessons relating to when to intervene 2. Lessons relating to improving judgmental estimation

  41. Lesson relating to when to intervene • Beware misconceptions of randomness, underweighting of base-rate information & recency biases -require recording of reasons for judgmental intervention • Beware overconfidence in judgment - restrict what-if analyses? • Beware numbers masquerading as forecasts • Build explanations into software – so its advice carries greater weight

  42. Lessons relating to improving judgmental estimation - Supporting the use of analogies • Involves identifying similar products, or similar promotion campaigns, and using these as a basis for the judgmental forecast E.g. We have a 2 week B.O.G.O.F promotion in the North region starting on 5 June 2006… Database identifies most similar cases:

  43. Supporting the use of analogies Support for different stages of judgmental process: • Memory support –so forecaster avoids need to recall past cases • Similarity support –helps forecaster to identify most similar past cases • Adaptation support -helps forecaster to adapt from similar past cases to specifics of current case Eg. -Most similar past BOGOF promotion may have lasted for 3 weeks, - Forthcoming BOGOF promotion lasts for only 2 weeks. - Database will estimate effect of 2 weeks rather than 3 from all past promotions…..

  44. Adaptation judgment support The interface Similarity judgment support Memory support

  45. Lessons relating to improving judgmental estimation • Using profiles to estimate effects of special events over time….. • E.g. timing effects of unexpected financial crisis in SE Asia

  46. Lessons relating to improving judgmental estimation • Using decomposition to reduce the demands of holistic estimation With some obvious macro analogies

  47. Macroeconomic forecasting errors Evidence of forecast failure (Fildes & Stekler, 2002) Cyclical turns They are missed Conservatism in underestimating growth in periods of expansion, overestimating in periods of contraction Inflation underestimated when accelerating, overestimating when slowing Rationality Observed inefficiencies Failure to improve over time Can the lessons learnt in micro forecasting help?

  48. Explaining the errors (Stekler, 2007) • Model derived errors • Structural change • Model and data interactions • Incompatibility between model / data vintages • Forecasters loss functions • Institutional effects • Judgemental effects • Bias • anchors

  49. Our focus: judgemental adjustments In macro forecasting, why are they made?(Donihue, J. Forecasting, 1993) • incompleteness in model • dotcom • structural change, • New exchange rate regime • missing variable in historic data base • SE Asia financial crisis • data inadequacies • Current data or revised data • recent ‘level’ errors

More Related