1 / 65

The Surprising Consequences of Randomness

The Surprising Consequences of Randomness. LS 829 Mathematics in Science and Civilization. Feb 6, 2010. Sources and Resources. Statistics: A Guide to the Unknown, 4 th ed., by R.Peck, et al. Publisher: Duxbury, 2006

jerome
Download Presentation

The Surprising Consequences of Randomness

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Surprising Consequences of Randomness LS 829 Mathematics in Science and Civilization Feb 6, 2010 LS 829 - 2010

  2. Sources and Resources • Statistics: A Guide to the Unknown, 4th ed., by R.Peck, et al. Publisher: Duxbury, 2006 • Taleb, N. N. (2008) Fooled by Randomness The Hidden Role of Chance in the Markets and Life, 2nd Edition. Random House. • Mlodinow, L (2008) The Drunkard’s Walk. Vintage Books. New York. • Rosenthal, J.S. (2005) Struck by Lightning Harper Perennial. Toronto. • www.stat.sfu.ca/~weldon LS 829 - 2010

  3. Introduction • Randomness concerns Uncertainty - e.g. Coin • Does Mathematics concern Certainty? - P(H) = 1/2 • Probability can help to Describe Randomness & “Unexplained Variability” • Randomness & Probability are key concepts for exploring implications of “unexplained variability” LS 829 - 2010

  4. Abstract Real World Mathematics Applications of Mathematics Probability Applied Statistics UsefulPrinciples Surprising Findings Nine Findings and Associated Principles LS 829 - 2010

  5. Example 1 - When is Success just Good Luck? An example from the world of Professional Sport LS 829 - 2010

  6. LS 829 - 2010

  7. LS 829 - 2010

  8. Sports League - FootballSuccess = Quality or Luck? LS 829 - 2010

  9. LS 829 - 2010

  10. Recent News Report “A crowd of 97,302 has witnessed Geelong break its 44-year premiership drought by crushing a hapless Port Adelaide by a record 119 points in Saturday's grand final at the MCG.” (2007 Season) LS 829 - 2010

  11. Sports League - FootballSuccess = Quality or Luck? LS 829 - 2010

  12. Are there better teams? • How much variation in the total points table would you expect IFevery team had the same chance of winning every game? i.e. every game is 50-50. • Try the experiment with 5 teams. H=Win T=Loss (ignore Ties for now) LS 829 - 2010

  13. 5 Team Coin Toss Experiment • Win=4, Tie=2, Loss=0 but we ignore ties. P(W)=1/2 • 5 teams (1,2,3,4,5) so 10 games as follows • 1-2,1-3,1-4,1-5,2-3,2-4,2-5,3-4,3-5,4-5 My experiment … • T T H T T H H H H T Experiment Result -----> But all teams Equal Quality (Equal Chance to win) LS 829 - 2010

  14. Implications? • Points spread due to chance? • Top team may be no better than the bottom team (in chance to win). LS 829 - 2010

  15. Simulation: 16 teams, equal chance to win, 22 games LS 829 - 2010

  16. Sports League - FootballSuccess = Quality or Luck? LS 829 - 2010

  17. Does it Matter? Avoiding foolish predictions Managing competitors (of any kind) Understanding the business of sport Appreciating the impact of uncontrolled variation in everyday life LS 829 - 2010

  18. Point of this Example? Need to discount “chance” In making inferences from everyday observations. LS 829 - 2010

  19. Example 2 - Order from Apparent Chaos An example from some personal data collection LS 829 - 2010

  20. Gasoline Consumption Each Fill - record kms and litres of fuel used Smooth ---> Seasonal Pattern …. Why? LS 829 - 2010

  21. Pattern Explainable? Air temperature? Rain on roads? Seasonal Traffic Pattern? Tire Pressure? Info Extraction Useful for Exploration of Cause Smoothing was key technology in info extraction LS 829 - 2010

  22. Intro to smoothing with context … STAT 100

  23. Optimal Smoothing Parameter? • Depends on Purpose of Display • Choice Ultimately Subjective • Subjectivity is a necessary part of good data analysis LS 829 - 2010

  24. Summary of this Example • Surprising? Order from Chaos … • Principle - Smoothing and Averaging reveal patterns encouraging investigation of cause LS 829 - 2010

  25. 3. Weather Forecasting LS 829 - 2010

  26. Chaotic Weather • 1900 – equations too complicated to solve • 2000 – solvable but still poor predictors • 1963 – The “Butterfly Effect” small changes in initial conditions -> large short term effects • today – ensemble forecasting see p 173 • Rupert Miller p 178 – stats for short term … LS 829 - 2010

  27. Conclusion from Weather Example? • It may not be true that weather forecasting will improve dramatically in the future • Some systems have inherent instability and increased computing power may not be enough the break through this barrier LS 829 - 2010

  28. Example 4 - Obtaining Confidential Information • How can you ask an individual for data on • Incomes • Illegal Drug use • Sex modes • …..Etc in a way that will get an honest response? There is a need to protect confidentiality of answers. LS 829 - 2010

  29. Example: Marijuana Usage • Randomized Response TechniquePose two Yes-No questions and have coin toss determine which is answeredHead 1. Do you use Marijuana regularly?Tail 2. Is your coin toss outcome a tail? LS 829 - 2010

  30. Randomized Response Technique • Suppose 60 of 100 answer Yes. Then about 50 are saying they have a tail. So 10 of the other 50 are users. 20%. • It is a way of using randomization to protect Privacy. Public Data banks have used this. LS 829 - 2010

  31. Summary of Example 4 • Surprising that people can be induced to provide sensitive information in public • The key technique is to make use of the predictability of certain empirical probabilities. LS 829 - 2010

  32. 5. Randomness in the Markets • 5A. Trends That Deceive • 5B. The Power of Diversification • 5C. Back-the-winner fallacy LS 829 - 2010

  33. 5A. Trends That Deceive People often fail to appreciate the effects of randomness LS 829 - 2010

  34. The Random Walk LS 829 - 2010

  35. Trends that do not persist LS 829 - 2010

  36. Longer Random Walk LS 829 - 2010

  37. Recent Intel Stock Price LS 829 - 2010

  38. Things to Note • The random walk has no patterns useful for prediction of direction in future • Stock price charts are well modeled by random walks • Advice about future direction of stock prices – take with a grain of salt! LS 829 - 2010

  39. 5B. The Power of Diversification People often fail to appreciate the effects of randomness LS 829 - 2010

  40. Preliminary Proposal I offer you the following “investment opportunity” You give me $100. At the end of one year, I will return an amount determined by tossing a fair coins twice, as follows: $0 ………25% of time (TT) $50.……. 25% of the time (TH) $100.……25% of the time (HT) $400.……25% of the time. (HH) Would you be interested? LS 829 - 2010

  41. Stock Market Investment • Risky Company - example in a known context • Return in 1 year for 1 share costing $10.00 25% of the time0.50 25% of the time1.00 25% of the time4.00 25% of the time i.e. Lose Money 50% of the time Only Profit 25% of the time “Risky” because high chance of loss LS 829 - 2010

  42. Independent Outcomes • What if you have the chance to put $1 into each of 100 such companies, where the companies are all in very different markets? • What sort of outcomes then? Use coin-tossing (by computer) to explore LS 829 - 2010

  43. Diversification Unrelated Companies • Choose 100 unrelated companies, each one risky like this. Outcome is still uncertain but look at typical outcomes …. One-Year Returns to a $100 investment LS 829 - 2010

  44. Looking at Profit only Avg Profit approx 38% LS 829 - 2010

  45. Gamblers like Averages and Sums! • The sum of 100 independent investments in risky companies is very predictable! • Sums (and averages) are more stable than the things summed (or averaged). • Square root law for variability of averages Variation -----> Variation/n LS 829 - 2010

  46. Summary - Diversification • Variability is not Risk • Stocks with volatile prices can be good investments • Criteria for Portfolio of Volatile Stocks • profitable on average • independence (or not severe dependence) LS 829 - 2010

  47. 5C - Back-the-winner fallacy • Mutual Funds - a way of diversifying a small investment • Which mutual fund? • Look at past performance? • Experience from symmetric random walk … LS 829 - 2010

  48. Implication from Random Walk …? • Stock market trends may not persist • Past might not be a good guide to future • Some fund managers better than others? • A small difference can result in a big difference over a long time … LS 829 - 2010

  49. A simulation experiment to determine the value of past performance data • Simulate good and bad managers • Pick the best ones based on 5 years data • Simulate a future 5-yrs for these select managers LS 829 - 2010

  50. How to describe good and bad fund managers? • Use TSX Index over past 50 years as a guide ---> annualized return is 10% • Use a random walk with a slight upward trend to model each manager. • Daily change positive with probability p LS 829 - 2010

More Related