Ameliorating mental mistakes in tradeoff studies
Download
1 / 79

Ameliorating Mental Mistakes in Tradeoff Studies - PowerPoint PPT Presentation


  • 296 Views
  • Uploaded on

Ameliorating Mental Mistakes in Tradeoff Studies. Terry Bahill Systems and Industrial Engineering University of Arizona [email protected] ©, 1993-2010, Bahill This file is located at http://www.sie.arizona.edu/sysengr/slides/. Acknowledgement. This research was supported by

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Ameliorating Mental Mistakes in Tradeoff Studies' - johana


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Ameliorating mental mistakes in tradeoff studies

Ameliorating Mental Mistakes in Tradeoff Studies

Terry Bahill

Systems and Industrial Engineering

University of Arizona

[email protected]

©, 1993-2010, Bahill

This file is located at http://www.sie.arizona.edu/sysengr/slides/


Acknowledgement
Acknowledgement

This research was supported by

AFOSR/MURI F49620-03-1-0377.


Reference
Reference

Smith, E. D., Son, Y. J., Piattelli-Palmarini, M. and Bahill, A. T., Ameliorating mental mistakes in tradeoff studies, Systems Engineering, 10:3, 222-240, 2007.

All of the material in this presentation is based on peer-reviewed journal papers. None of it comes from the Internet.


Present situation
Present situation

  • Tradeoff studies are broadly recognized by

    CMMI and recommended as a Decision Analysis and Resolution (DAR) method for simultaneously considering multiple alternatives with many criteria.

  • Tradeoff studies, which involve human

    • calibration

    • data updating

    • numerical judgment,

  • are often muddled by analysts

  • are often distrusted by decision makers.


  • Resolution
    Resolution

    • The decision-making fields of

      • Judgment and Decision Making

      • Cognitive Science

      • Experimental Economics

        have a large body of research on human biases and errors in considering numerical judgments and criteria-based choices.

  • Similarities between their experiments and the elements of tradeoff studies show that tradeoff studies are susceptible to human biases.


  • Nobel prize
    Nobel Prize

    Daniel Kahneman won the Nobel Prize in Economics in 2002 "for having integrated insights from psychological research into economic science, especially concerning human judgment and decision-making under uncertainty."


    Judgment and decision making experiments
    Judgment and decision making experiments

    • Allais paradox

    • Thaler paradox

    • Ellsberg paradox

    • Reflection effect

    • Certainty effect

    • Law of small numbers

    • Ranking in subjective probability

    • Strength and weight

    • Value versus Utility

    • Probabilities

    • Risks and uncertainties

    • Prospects

    • Time discounting

    • Elimination by aspects


    Our goal
    Our goal

    • We want to help people create tradeoff studies to choose among alternatives.

    • We want people to have confidence that they made the right decision.

    • We recommend actions that will help people avoid making specific mental mistakes in doing tradeoff studies.

    • These recommendations are the prime deliverable of this research effort.


    Eric Smith studied hundreds of experimental papers and isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.


    Components of a tradeoff study
    Components of a tradeoff study isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Problem statement

    • Evaluation criteria

    • Weights of importance

    • Alternative solutions

    • Evaluation data

    • Scoring functions

    • Normalized scores

    • Combining functions

    • Preferred alternatives

    • Sensitivity analysis


    Mental mistakes
    Mental mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Emotions, cognitive illusions, conscious and unconscious biases, fallacies, fear of regret and the use of heuristics can cause mistakes in tradeoff studies.

    • We will group all these terms under the phrase mental mistakes.

    • The following four dozen slides list specific mental mistakes and state how they can affect particular components of tradeoff studies.


    Problem statement mistakes
    Problem Statement Mistakes* isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Bad problem stating

    • Incorrect phrasing

    • Ambiguous problem stating

    • Substituting a related attribute

    • Feeling invincible


    Bad problem stating
    Bad problem stating isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.^

    • “The problem of the design of a system must be stated strictly in terms of its requirements, not in terms of a solution or a class of solutions.” Wayne Wymore

    • It is a mistake to state the problem in terms of a solution instead of the customer needs and expectations.

    • Recommendation: Communicate with and question the customer in order to determine his or her values and needs.


    Incorrect phrasing
    Incorrect phrasing isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Phrasing of the question affects the answer

      • Problem-M: Several Australian mammal species are nearly wiped out by hunters.Intervention: Contribute to a fund to provide a safe breeding area for these species.

      • Problem-W: Skin cancer from sun exposure is common among farm workers.Intervention: Support free medical checkups for threatened groups.

    • When asked about giving money, subjects said they would contribute more money to provide a safe breeding area than for free medical checkups.

    • However, when asked which intervention they would support, they said they would rather support free medical checkups.

    • Recommendation: Questions designed to get a value for a criterion should be tightly coupled to the criterion.


    Phrasing
    Phrasing isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.*

    • The way you phrase the question will determine the answer you get.

    • When asked whether they would approve surgery in a hypothetical medical emergency, many more people accepted surgery when the chance of survival was given as 99 percent than when the chance of death was given as 1 percent.


    Preference reversals

    $ bet isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    Has higher dollar value

    P bet

    Has higher probability

    Preference reversals*

    Although the expected values are the same,

    most people preferred to play the P bet, however

    most people wanted a higher selling price for the $ bet.


    Ambiguous problem stating
    Ambiguous problem stating isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.^

    • If a problem statement is vague (such as “work for the public good”) then proposed solutions can vary greatly, and derive support for very different reasons and in different ways.

    • Recommendation: State the problem without ambiguity; which is more ambiguous (1) to allocate physical resources or (2) to influence perceptions through psychology?


    Substituting a simpler process
    Substituting a simpler process isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Sometimes a person substitutes a related entity that comes to mind more readily. In effect, “people who are confronted with a difficult question sometimes answer an easier one instead.”

    • When making a decision that should be decided by a tradeoff study, people sometimes substitute a simpler decision process.

    • Recommendation: Decision makers should realize that a premature reduction of a tradeoff study to a simpler decision process is a common heuristic that prevents through consideration of the original decision.


    Feeling invincible
    Feeling invincible isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.*

    • Teen-age boys are notorious for thinking

      • I won’t get caught

      • I can’t get hurt

      • I will avoid car accidents

      • I won’t cause an unwanted pregnancy

      • I won’t get sexually transmitted disease

      • I don’t have to back up my hard drive, my computer won’t crash

      • They can’t do that to me

    • In 1912, the White Star line said that the Titanic was ‘unsinkable.’

    • Recommendation: Decision makers must learn and have the freedom to question statements that are obviously true and other sacred cows.


    Feeling invincible 2
    Feeling invincible isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.2

    • Codebreakers have been routinely breaking codes for over 600 years.

    • During WWII the American generals often had copies of Hitler’s battle orders before the German generals.

    • Yet the Americans did not think that the Germans were breaking the American codes. (I do not know for a fact that they were.)


    Evaluation criteria mistakes
    Evaluation Criteria Mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Dependent criteria

    • Relying on personal experience

    • Forer Effect


    Dependent criteria
    Dependent criteria isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.*

    • Evaluation criteria should be independent.^

    • For evaluating humans, Height and Weight are not independent: Sex (male versus female) and Intelligence Quotient are independent.

    • Recommendation: Dependent criteria should be grouped together as subcriteria.


    Relying on personal experience
    Relying on personal experience isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • "We are all prisoners of our own experience.”

    • Criteria may be chosen by the analyst's experience, with insufficient customer input and environmental confirmation.

    • Recommendation: It is imperative to conduct thorough searches for objective knowledge. Talk to your customer and other stakeholders.


    Forer effect
    Forer effect isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • The analyst might fail to question or re-write criteria from a legacy tradeoff study that originated from a perceived authority and is now seemingly adaptable to the tradeoff at hand.

    • Recommendations:

      • Give some time to considering and formulating criteria from scratch, before consulting and possibly reusing previously written criteria.

      • Generic criteria taken from the company process assets library must be tailored for the project at hand.


    Weight of importance mistakes
    Weight of Importance Mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Choice versus calculation

    • Ignoring severity amplifiers


    Choice versus calculation
    Choice versus calculation isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    Choice, 67% chose Program X

    Calculation, 4% calculated $55M or more


    Ignoring severity amplifiers
    Ignoring severity amplifiers isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.*

    Different people will give different weights of importance because of their perceptions of

    Recommendation: Intersubject variability can be reduced with education, peer review of the assigned weights and group discussions. Keep a broad view of the whole organization, so that criteria in one area are considered in light of all others.


    Alternative solution mistakes
    Alternative Solution Mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Serial consideration of alternatives

    • Isolated or juxtaposed alternatives

    • Conflicting criteria

    • Adding alternatives

    • Maintaining the status quo

    • Uneven level of detail


    Serial consideration of alternatives
    Serial consideration of alternatives isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • When solving a problem, people seize on a hypothesis and hold on to it until it is disproved.

    • Once the hypothesis is disproved, they will progress to the next hypothesis and hold on to it until it is disproved.

    • This bias can persist throughout a tradeoff study, as an analyst uses the whole study to try to prove that a currently favored alternative is the best.

    • Recommendation: Alternative solutions should be evaluated in parallel from the beginning of the tradeoff study, so that a collective and impartial consideration will permit the selection of the best alternative from a complete solution space.


    Isolated or juxtaposed alternatives
    Isolated or juxtaposed alternatives isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Two dictionaries were evaluated in isolation and juxtaposed.

    • When evaluated in isolation, subjects were willing to pay more for dictionary A than for B. However, when evaluated at the same time, subjects were willing to pay more for dictionary B.

    • Recommendations:

      • New alternative solutions should be

        subject to elimination only after

        comparison to all alternative solutions.

      • Group alternatives by affinities.


    Conflicting criteria
    Conflicting criteria isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    “You can either select one of these gambles or you can pay $1 to add one more gamble to the choice set. The added gamble will be selected at random from the list you reviewed.”


    Adding alternatives
    Adding alternatives isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Patient M. S. is a 52-year-old journalist with a mini-stroke. She had a similar episode ten days ago that lasted about 12 hours. Angiography shows a 70% constriction of the left carotid artery. Past medical history is noteworthy for past alcoholism (no liver cirrhosis) and mild diabetes (diet controlled)

    • Patient A. R. is a 72-year-old retired police officer with a mini-stroke. He had two similar episodes in the last three months with the last occurring one month ago. Angiography shows a 90% constriction of the right carotid artery. He has no concurrent medical problems and is in generally good health.

    • On which patient would you operate first? 38% of the physicians chose Patient A. R.


    The additional alternative
    The additional alternative isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Patient P. K. is a 55-year-old bartender with a mini-stroke. She had one similar episode a week ago that lasted about 6 hours. Angiography shows a 75% constriction of the ipsilateral carotid artery. Past medical history is noteworthy for ongoing cigarette smoking (since age 15 at a rate of one pack per day).

    • In the group of deciders that was given all three patients, 58% of the physicians now chose Patient A. R., a big increase.

    • Recommendation:

      • All of the alternative solutions should be evaluated in parallel from the beginning of the tradeoff study.

      • If an alternative must be added in the middle of a study, then the most similar alternative will lose support.


    Maintaining the status quo
    Maintaining the status quo isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Students were paid $1.50.

    • Then they were asked to trade their $1.50 for a metal Zebra pen: 25% kept the $1.50.

    • Then they were asked to trade their $1.50 for either a metal Zebra pen or two plastic Pilot pens: 53% kept the $1.50.

    • An increase in the conflict of the choice increased their decision to stay with the status quo.

    • Recommendation: Do not needlessly increase the number of alternatives.


    Uneven level of detail
    Uneven level of detail isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Uneven level of detail in the description of the alternatives might confuse a naive reader.

    • If alternatives are abstracted at a different level of detail it will be difficult to assign scores to the alternatives.


    Evaluation data mistakes
    Evaluation Data Mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Relying on personal experience

    • Magnitude and reliability

    • Judging probabilities poorly


    Relying on personal experience1
    Relying on personal experience isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Estimates for evaluation data may faultily come from personal experiences.

    • People may be completely oblivious to things they have not experienced, or they may think that their limited experience is complete.

    • What people think they know may be different from what they actually know.

    • Recommendations:

      • The source of evaluation data must be subject to peer and public review.

      • Decision analysts must be willing to yield absolute control over evaluation data.


    Magnitude and reliability
    Magnitude and reliability isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • People tend to judge the validity of data first on its magnitude (‘strength’), and then according to its reliability (‘weight’).

    • Therefore, data with outstanding magnitudes but poor reliability are likely to be chosen and used.

    • Recommendation: Either data with uniform reliability should be used, or the speciousness of data should be taken into account in the Risk portion of a tradeoff study.


    Humans judge probabilities poorly
    Humans judge probabilities poorly isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.*


    Probabilistic illusions

    Gambler’s fallacy isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    Over-Alternation fallacy

    Conjunction fallacy

    Disjunction fallacy

    Law of small numbers

    Extensionality fallacies

    Mis-Estimation of probabilities

    Ease of Representation: Typicality

    Sub-Additively

    Super-Additively

    Confirmation bias

    Certainty effect

    Ambiguity aversion

    Aversion to sequences of chance events

    Delay-Speedup asymmetry

    Loss/Gain discounting

    Frequency Illusions

    Base-Rate Neglect

    Probabilistic illusions


    Ignoring the first measurement 1
    Ignoring the first measurement isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.1

    • Often when a measurement (test) reveals an unexpected result, the physician and/or the patient will ask for a second measurement.

    • If the second measurement is pleasing, then the first measurement is discarded and only the result of the last measurement is recorded.

    • Recommendation: If there is no evidence showing why the first measurement was in error, then it should not be discarded.


    Ignoring the first measurement 2
    Ignoring the first measurement isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.2

    • A reasonable strategy would be to record the average of the two measurements.

    • For example, if you take your blood pressure and the result is abnormally high, then you might measure it again.

    • If the second measurement indicates that blood pressure is in the normal range, and you do not have proof that the first reading was a mistake, then do not record only the second

      reading, either record both

      measurements or the average

      of the two readings.


    Scoring function mistakes
    Scoring Function Mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Mixing gains and losses

    • Not using scoring functions

    • Anchoring


    Scoring functions
    Scoring functions isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Objective value is translated to subjective worth

    • Input values become normalized output scores

    • Scoring functions must be elicited from the customer


    Gains and losses are not equal
    Gains and losses are not equal isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.*


    Percent happy scouts mistake
    Percent happy scouts mistake isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • The Pinewood Derby tradeoff study had these criteria

      • Percent Happy Scouts

      • Number of Irate Parents

    • Because people evaluate losses and gains differently, the Preferred alternatives might have been different if they had used

      • Percent Unhappy Scouts

      • Number of Ecstatic Parents


    Recommendation: isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet. Scoring functions in a tradeoff study

    should express gains rather than losses.


    Not using scoring functions
    Not using scoring functions isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Most tradeoff studies that we have observed in industry did not use scoring functions.

    • In some cases, scoring functions were explained in the company’s engineering process, but they were not convenient, hence they were not used.

    • Recommendation: The Wymorian standard scoring functions should be used in tradeoff studies. Those located at http://www.sie.arizona.edu/sysengr/slides/,

      should be referenced in company engineering processes.


    Anchoring
    Anchoring isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • A person’s first impression dominates all further thought.

    • People were shown a wheel of fortune with numbers from one to hundred.

    • The wheel was spun and the subjects were asked to estimate the number of African nations in the United Nations.

    • If the wheel showed a small number, like 12, the subjects underestimated the correct number.

    • If the wheel showed a large number, like 92, the subjects overestimated the correct number.

    • Recommendation: When estimating values for parameters of scoring functions, think about the whole range of expected values for the parameters.


    Anchoring 2
    Anchoring isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.2

    • You should fill out a tradeoff study matrix row by row with the status quo as the first alternative. Therefore, the values of the status quo are the anchors for estimating the other data. Unfortunately, the status quo is likely to have extremely low values for performance and extremely high values for cost, schedule and risk. But at least the anchoring alternative is known, consistent and you have control over it.

    • Recommendations:

      • Make the status quo the first alternative.

      • In one iteration examine the scores left to right and in the next iteration examine them right to left.


    Output score mistakes
    Output Score Mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • False precision


    False precision
    False precision isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • The most common mistake in tradeoff studies is false precision. For example, a tradeoff analyst asks an expert to estimate values for two criteria. The expert says, “The first criterion is about 2 and the second is around 3.” The analyst puts these numbers into a calculator and computes the ratio as 0.666666667. This is nonsense, but these nine significant digits are dragged throughout the tradeoff study. The Forer Effect might explain this: the analyst believes that the calculator is an impeccable authority in calculating numbers. Therefore, what the calculator says must be true.

    • Recommendation: In numerical tables, print only the number of digits after the decimal place that are necessary to show a difference between the preferred alternatives.


    Combining function mistakes
    Combining Function Mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Lack of knowledge

    • Lack of availability


    Lack of knowledge
    Lack of knowledge isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • The average engineer is not familiar with the nuances of combining functions and their behavior specific to tradeoff studies.

    • Recommendation: Training with combining functions is necessary.


    Unavailability
    Unavailability isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Software is equipped with limited types of combining functions.

    • For example, one of the best commercial tools, Expert Choice, has only the Sum and the Product combining functions.

    • Most others have only the Sum.

    • Recommendation: Spreadsheet-formulated tradeoff studies have the greatest potential for combining function variety.


    Popular combining functions
    Popular combining functions isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Sum Combining Function = x + y

      • Used most often by engineers

    • Product Combining Function = x y

      • Cost to benefit ratio

      • Risk analyses

      • Game theory*

    • Sum Minus Product = x + y - xy

      • Probability theory

      • Fuzzy logic systems

      • Expert system certainty factors

    • Compromise =


    Summation is not always the best way to combine data
    Summation is not always isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.the best way to combine data*


    Preferred alternative mistakes
    Preferred Alternative Mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Overconfidence

    • Ignoring the need for expert opinion

    • Failure to talk with the customer


    Overconfidence
    Overconfidence isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Tradeoff studies are often started with over confidence.

    • The analyst prefers to maintain a state of over confidence without examining details.

    • Recommendation: For this bias, there is no better teacher than performing tradeoff studies, and bringing subjects to reviews and customer acceptance that require producing high-quality work in all tradeoff study components.


    Obviating expert opinion
    Obviating expert opinion isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • The analyst holds a circular belief that expert opinion or review is not necessary because no evidence for the need of expert opinion is present.

    • Recommendation: Experts should be sought formally or informally to evaluate tradeoff study work.

    • In the last 40 years, the most common student mistake that Bahill has observed is failure to seek advice from experts and advisors.^


    Failure to talk with the customer
    Failure to talk with the customer isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • The second most common mistake is failing to talk with the customer. Students and engineers seem to feel that it is a sign of weakness to ask for help.

    • Recommendation. Talk with your customer.


    Sensitivity analysis mistakes
    Sensitivity Analysis Mistakes isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Lack of training

    • Hawthorne effect


    Lack of training
    Lack of training isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Most engineers are not trained in sensitivity analyses.

    • Interactions among parameters can be very important.

    • Step sizes for the approximation of effects should be very small.

    • Second-order derivatives must be calculated accurately.

    • Recommendation: Investments in sensitivity analysis training must be made. Perhaps enabling software can substitute for much sensitivity analysis knowledge.


    Hawthorne effect
    Hawthorne effect isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.*

    • If you paint the factory walls a bright color, productivity goes up.

    • If you increase the illumination, productivity goes up.

    • If you decrease the illumination, productivity goes up.

    • Recommendation:

    • Maybe Hawthorne (and the Heisenberg Uncertainty Principle) affect evaluation data. If you measure something in a system, then its performance is likely to improve.


    Discussion isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.


    Your job
    Your job isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    is to help a decision maker make valid decisions that he or she (and other stakeholders) will have confidence in.

    • This is a difficult and iterative task.

    • It entails discovering the decision makers preferred weights, scoring functions, and combining functions.

    • You must also discover his or her mental mistakes and ameliorate them.

    • You must get into the head of the decision maker and discover his or her values*


    Personality types
    Personality types isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Different people have different personality types.

    • The Myers-Briggs model is one way of describing these personality types.

    • Sensory - Thinking – Judging people are likely to appreciate the tradeoff study techniques we have presented.

    • Intuitive – Feeling – Perceiving people most likely will not.


    Factors affecting human decisions
    Factors affecting human decisions isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • the decision maker

      • corporate culture

      • the decision maker’s values

      • personality types

      • risk averseness

      • mental mistakes*

    • information display

      • wording of the question

      • context of presentation

    • the decision

      • effort required to make the decision

      • difficulty of making the decision

      • time allowed to make the decision

      • needed accuracy of the decision

      • cost of the decision

      • likelihood of regret


    Good industry practices
    Good industry practices isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    for ensuring success of tradeoff studies include

    • having teams evaluate the data

    • evaluating the data in many iterations

    • expert review of the results and recommendations*


    Purpose of teaching tradeoff studies
    Purpose of teaching tradeoff studies isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Emotions, illusions, biases and use of heuristics make humans far from ideal decision makers.

    • Using tradeoff studies thoughtfully can help move your decisions from the normal human decision-making lower-right

      quadrant to the ideal

      decision-making

      upper-left quadrant.


    Improving the tradeoff process
    Improving the tradeoff process isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    • Inform decision makers about how mental mistakes affect tradeoff studies, forewarned is forearmed

    • Creating a long-term, institutional decision horizon usually increases rationality

    • Team approach

    • Iterations

    • Public reviews

    • Reduce mental errors by using the recommendations given in this presentation


    Pr cis
    Pr isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.écis

    • Tradeoff studies seek to build a mathematical framework

    • The goal is a correct, parallel mathematical consideration of all relevant criteria, avoiding misjudgments associated with the serial consideration of criteria in subgroups

    • Cognitive science can help improve the validity and sensitivity of tradeoff studies


    Rms titanic 1912
    RMS Titanic, 1912 isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.


    The rms titanic lifeboat decision
    The RMS Titanic lifeboat decision isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.

    The original design for the RMS Titanic called for 64 lifeboats, but this was reduced to 20 before its maiden voyage: this might have been a mistake. The Chief Designer (CD) wanted 64 lifeboats. But the Program Manager (PM) reduced it to 20 after his advisors told him only 16 were required by law. The CD resigned over this decision. The British Board of Trade regulations of 1894 specified the lifeboat capacity. For ships over 10,000 tons, this lifeboat capacity was specified by volume (5,500 cubic feet), which could be converted into passenger seats (about 1000) or the number of lifeboats (about 16). So, even though the Titanic displaced 46,000 tons and was certified to carry 3,500 passengers, its 20 lifeboats complied with the regulations of the time. But let us go back to the design decision to reduce the number of lifeboats from 64 to 20. What if they had performed the following hypothetical tradeoff study? In this table, the weights of importance range from 0 to 10, with 10 being the most important and the evaluation data (scores) also range from 0 to 10, with 10 being the best. For simplicity, we have not used scoring functions, so the evaluation data are also the scores.


    Mental mistakes in the titanic decision 1
    Mental mistakes in the Titanic decision isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.1

    • The Program Manager and Chief Designer had different preferred alternatives because of their different weights of importance.

    • The PM had overconfidence in his subjective choice of 20 lifeboats. If he had done this tradeoff study, might he have rethought his decision?

    • In 1912, the White Star line said the Titanic was “unsinkable.” If the PM did not feel invincible, would he have authorized more lifeboats?

    • If the PM understood the Forer effect (that an analyst might fail to question or re-write criteria that originated from a perceived authority), might he have reassessed the Board of Trade’s requirement for 16 lifeboats?


    Mental mistakes in the titanic decision 2
    Mental mistakes in the Titanic decision isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.2

    • The PM and the CD did not do a tradeoff study.

    • They merely discussed the 20 and 64-lifeboat alternatives.

    • If they had understood distinctiveness by the addition of alternatives and had done this tradeoff study with the addition of the 10 and 30-lifeboat alternatives, would the PM have chosen a different alternative?


    Mental mistakes in the titanic decision 3
    Mental mistakes in the Titanic decision isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.3

    • A sensitivity analysis of the tradeoff study shows that the PM’s most important parameter is the weight of importance for the Cost criterion and that the CD’s most important parameter is the weight of importance for the Percentage of People that Could be Accommodated criterion.

    • Therefore, the PM should have spent more time assessing the magnitude and reliability of these weights.

    • In fact, he should have noted the importance of lifeboat regulations, and questioned whether such regulations were up-to-date for the new, larger Titanic design.


    ad