ameliorating mental mistakes in tradeoff studies
Download
Skip this Video
Download Presentation
Ameliorating Mental Mistakes in Tradeoff Studies

Loading in 2 Seconds...

play fullscreen
1 / 79

Ameliorating Mental Mistakes in Tradeoff Studies - PowerPoint PPT Presentation


  • 297 Views
  • Uploaded on

Ameliorating Mental Mistakes in Tradeoff Studies. Terry Bahill Systems and Industrial Engineering University of Arizona [email protected] ©, 1993-2010, Bahill This file is located at http://www.sie.arizona.edu/sysengr/slides/. Acknowledgement. This research was supported by

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Ameliorating Mental Mistakes in Tradeoff Studies' - johana


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
ameliorating mental mistakes in tradeoff studies

Ameliorating Mental Mistakes in Tradeoff Studies

Terry Bahill

Systems and Industrial Engineering

University of Arizona

[email protected]

©, 1993-2010, Bahill

This file is located at http://www.sie.arizona.edu/sysengr/slides/

acknowledgement
Acknowledgement

This research was supported by

AFOSR/MURI F49620-03-1-0377.

reference
Reference

Smith, E. D., Son, Y. J., Piattelli-Palmarini, M. and Bahill, A. T., Ameliorating mental mistakes in tradeoff studies, Systems Engineering, 10:3, 222-240, 2007.

All of the material in this presentation is based on peer-reviewed journal papers. None of it comes from the Internet.

present situation
Present situation
  • Tradeoff studies are broadly recognized by

CMMI and recommended as a Decision Analysis and Resolution (DAR) method for simultaneously considering multiple alternatives with many criteria.

  • Tradeoff studies, which involve human
      • calibration
      • data updating
      • numerical judgment,
  • are often muddled by analysts
  • are often distrusted by decision makers.
resolution
Resolution
  • The decision-making fields of
      • Judgment and Decision Making
      • Cognitive Science
      • Experimental Economics

have a large body of research on human biases and errors in considering numerical judgments and criteria-based choices.

  • Similarities between their experiments and the elements of tradeoff studies show that tradeoff studies are susceptible to human biases.
nobel prize
Nobel Prize

Daniel Kahneman won the Nobel Prize in Economics in 2002 "for having integrated insights from psychological research into economic science, especially concerning human judgment and decision-making under uncertainty."

judgment and decision making experiments
Judgment and decision making experiments
  • Allais paradox
  • Thaler paradox
  • Ellsberg paradox
  • Reflection effect
  • Certainty effect
  • Law of small numbers
  • Ranking in subjective probability
  • Strength and weight
  • Value versus Utility
  • Probabilities
  • Risks and uncertainties
  • Prospects
  • Time discounting
  • Elimination by aspects
our goal
Our goal
  • We want to help people create tradeoff studies to choose among alternatives.
  • We want people to have confidence that they made the right decision.
  • We recommend actions that will help people avoid making specific mental mistakes in doing tradeoff studies.
  • These recommendations are the prime deliverable of this research effort.
slide9
Eric Smith studied hundreds of experimental papers and isolated seven dozen biases that could affect the components of tradeoff studies. His results are summarized in this Excel spreadsheet.
components of a tradeoff study
Components of a tradeoff study
  • Problem statement
  • Evaluation criteria
  • Weights of importance
  • Alternative solutions
  • Evaluation data
  • Scoring functions
  • Normalized scores
  • Combining functions
  • Preferred alternatives
  • Sensitivity analysis
mental mistakes
Mental mistakes
  • Emotions, cognitive illusions, conscious and unconscious biases, fallacies, fear of regret and the use of heuristics can cause mistakes in tradeoff studies.
  • We will group all these terms under the phrase mental mistakes.
  • The following four dozen slides list specific mental mistakes and state how they can affect particular components of tradeoff studies.
problem statement mistakes
Problem Statement Mistakes*
  • Bad problem stating
  • Incorrect phrasing
  • Ambiguous problem stating
  • Substituting a related attribute
  • Feeling invincible
bad problem stating
Bad problem stating^
  • “The problem of the design of a system must be stated strictly in terms of its requirements, not in terms of a solution or a class of solutions.” Wayne Wymore
  • It is a mistake to state the problem in terms of a solution instead of the customer needs and expectations.
  • Recommendation: Communicate with and question the customer in order to determine his or her values and needs.
incorrect phrasing
Incorrect phrasing
  • Phrasing of the question affects the answer
    • Problem-M: Several Australian mammal species are nearly wiped out by hunters.Intervention: Contribute to a fund to provide a safe breeding area for these species.
    • Problem-W: Skin cancer from sun exposure is common among farm workers.Intervention: Support free medical checkups for threatened groups.
  • When asked about giving money, subjects said they would contribute more money to provide a safe breeding area than for free medical checkups.
  • However, when asked which intervention they would support, they said they would rather support free medical checkups.
  • Recommendation: Questions designed to get a value for a criterion should be tightly coupled to the criterion.
phrasing
Phrasing*
  • The way you phrase the question will determine the answer you get.
  • When asked whether they would approve surgery in a hypothetical medical emergency, many more people accepted surgery when the chance of survival was given as 99 percent than when the chance of death was given as 1 percent.
preference reversals
$ bet

Has higher dollar value

P bet

Has higher probability

Preference reversals*

Although the expected values are the same,

most people preferred to play the P bet, however

most people wanted a higher selling price for the $ bet.

ambiguous problem stating
Ambiguous problem stating^
  • If a problem statement is vague (such as “work for the public good”) then proposed solutions can vary greatly, and derive support for very different reasons and in different ways.
  • Recommendation: State the problem without ambiguity; which is more ambiguous (1) to allocate physical resources or (2) to influence perceptions through psychology?
substituting a simpler process
Substituting a simpler process
  • Sometimes a person substitutes a related entity that comes to mind more readily. In effect, “people who are confronted with a difficult question sometimes answer an easier one instead.”
  • When making a decision that should be decided by a tradeoff study, people sometimes substitute a simpler decision process.
  • Recommendation: Decision makers should realize that a premature reduction of a tradeoff study to a simpler decision process is a common heuristic that prevents through consideration of the original decision.
feeling invincible
Feeling invincible*
  • Teen-age boys are notorious for thinking
    • I won’t get caught
    • I can’t get hurt
    • I will avoid car accidents
    • I won’t cause an unwanted pregnancy
    • I won’t get sexually transmitted disease
    • I don’t have to back up my hard drive, my computer won’t crash
    • They can’t do that to me
  • In 1912, the White Star line said that the Titanic was ‘unsinkable.’
  • Recommendation: Decision makers must learn and have the freedom to question statements that are obviously true and other sacred cows.
feeling invincible 2
Feeling invincible2
  • Codebreakers have been routinely breaking codes for over 600 years.
  • During WWII the American generals often had copies of Hitler’s battle orders before the German generals.
  • Yet the Americans did not think that the Germans were breaking the American codes. (I do not know for a fact that they were.)
evaluation criteria mistakes
Evaluation Criteria Mistakes
  • Dependent criteria
  • Relying on personal experience
  • Forer Effect
dependent criteria
Dependent criteria*
  • Evaluation criteria should be independent.^
  • For evaluating humans, Height and Weight are not independent: Sex (male versus female) and Intelligence Quotient are independent.
  • Recommendation: Dependent criteria should be grouped together as subcriteria.
relying on personal experience
Relying on personal experience
  • "We are all prisoners of our own experience.”
  • Criteria may be chosen by the analyst's experience, with insufficient customer input and environmental confirmation.
  • Recommendation: It is imperative to conduct thorough searches for objective knowledge. Talk to your customer and other stakeholders.
forer effect
Forer effect
  • The analyst might fail to question or re-write criteria from a legacy tradeoff study that originated from a perceived authority and is now seemingly adaptable to the tradeoff at hand.
  • Recommendations:
    • Give some time to considering and formulating criteria from scratch, before consulting and possibly reusing previously written criteria.
    • Generic criteria taken from the company process assets library must be tailored for the project at hand.
weight of importance mistakes
Weight of Importance Mistakes
  • Choice versus calculation
  • Ignoring severity amplifiers
choice versus calculation
Choice versus calculation

Choice, 67% chose Program X

Calculation, 4% calculated $55M or more

ignoring severity amplifiers
Ignoring severity amplifiers*

Different people will give different weights of importance because of their perceptions of

Recommendation: Intersubject variability can be reduced with education, peer review of the assigned weights and group discussions. Keep a broad view of the whole organization, so that criteria in one area are considered in light of all others.

alternative solution mistakes
Alternative Solution Mistakes
  • Serial consideration of alternatives
  • Isolated or juxtaposed alternatives
  • Conflicting criteria
  • Adding alternatives
  • Maintaining the status quo
  • Uneven level of detail
serial consideration of alternatives
Serial consideration of alternatives
  • When solving a problem, people seize on a hypothesis and hold on to it until it is disproved.
  • Once the hypothesis is disproved, they will progress to the next hypothesis and hold on to it until it is disproved.
  • This bias can persist throughout a tradeoff study, as an analyst uses the whole study to try to prove that a currently favored alternative is the best.
  • Recommendation: Alternative solutions should be evaluated in parallel from the beginning of the tradeoff study, so that a collective and impartial consideration will permit the selection of the best alternative from a complete solution space.
isolated or juxtaposed alternatives
Isolated or juxtaposed alternatives
  • Two dictionaries were evaluated in isolation and juxtaposed.
  • When evaluated in isolation, subjects were willing to pay more for dictionary A than for B. However, when evaluated at the same time, subjects were willing to pay more for dictionary B.
  • Recommendations:
    • New alternative solutions should be

subject to elimination only after

comparison to all alternative solutions.

    • Group alternatives by affinities.
conflicting criteria
Conflicting criteria

“You can either select one of these gambles or you can pay $1 to add one more gamble to the choice set. The added gamble will be selected at random from the list you reviewed.”

adding alternatives
Adding alternatives
  • Patient M. S. is a 52-year-old journalist with a mini-stroke. She had a similar episode ten days ago that lasted about 12 hours. Angiography shows a 70% constriction of the left carotid artery. Past medical history is noteworthy for past alcoholism (no liver cirrhosis) and mild diabetes (diet controlled)
  • Patient A. R. is a 72-year-old retired police officer with a mini-stroke. He had two similar episodes in the last three months with the last occurring one month ago. Angiography shows a 90% constriction of the right carotid artery. He has no concurrent medical problems and is in generally good health.
  • On which patient would you operate first? 38% of the physicians chose Patient A. R.
the additional alternative
The additional alternative
  • Patient P. K. is a 55-year-old bartender with a mini-stroke. She had one similar episode a week ago that lasted about 6 hours. Angiography shows a 75% constriction of the ipsilateral carotid artery. Past medical history is noteworthy for ongoing cigarette smoking (since age 15 at a rate of one pack per day).
  • In the group of deciders that was given all three patients, 58% of the physicians now chose Patient A. R., a big increase.
  • Recommendation:
    • All of the alternative solutions should be evaluated in parallel from the beginning of the tradeoff study.
    • If an alternative must be added in the middle of a study, then the most similar alternative will lose support.
maintaining the status quo
Maintaining the status quo
  • Students were paid $1.50.
  • Then they were asked to trade their $1.50 for a metal Zebra pen: 25% kept the $1.50.
  • Then they were asked to trade their $1.50 for either a metal Zebra pen or two plastic Pilot pens: 53% kept the $1.50.
  • An increase in the conflict of the choice increased their decision to stay with the status quo.
  • Recommendation: Do not needlessly increase the number of alternatives.
uneven level of detail
Uneven level of detail
  • Uneven level of detail in the description of the alternatives might confuse a naive reader.
  • If alternatives are abstracted at a different level of detail it will be difficult to assign scores to the alternatives.
evaluation data mistakes
Evaluation Data Mistakes
  • Relying on personal experience
  • Magnitude and reliability
  • Judging probabilities poorly
relying on personal experience1
Relying on personal experience
  • Estimates for evaluation data may faultily come from personal experiences.
  • People may be completely oblivious to things they have not experienced, or they may think that their limited experience is complete.
  • What people think they know may be different from what they actually know.
  • Recommendations:
    • The source of evaluation data must be subject to peer and public review.
    • Decision analysts must be willing to yield absolute control over evaluation data.
magnitude and reliability
Magnitude and reliability
  • People tend to judge the validity of data first on its magnitude (‘strength’), and then according to its reliability (‘weight’).
  • Therefore, data with outstanding magnitudes but poor reliability are likely to be chosen and used.
  • Recommendation: Either data with uniform reliability should be used, or the speciousness of data should be taken into account in the Risk portion of a tradeoff study.
probabilistic illusions
Gambler’s fallacy

Over-Alternation fallacy

Conjunction fallacy

Disjunction fallacy

Law of small numbers

Extensionality fallacies

Mis-Estimation of probabilities

Ease of Representation: Typicality

Sub-Additively

Super-Additively

Confirmation bias

Certainty effect

Ambiguity aversion

Aversion to sequences of chance events

Delay-Speedup asymmetry

Loss/Gain discounting

Frequency Illusions

Base-Rate Neglect

Probabilistic illusions
ignoring the first measurement 1
Ignoring the first measurement1
  • Often when a measurement (test) reveals an unexpected result, the physician and/or the patient will ask for a second measurement.
  • If the second measurement is pleasing, then the first measurement is discarded and only the result of the last measurement is recorded.
  • Recommendation: If there is no evidence showing why the first measurement was in error, then it should not be discarded.
ignoring the first measurement 2
Ignoring the first measurement2
  • A reasonable strategy would be to record the average of the two measurements.
  • For example, if you take your blood pressure and the result is abnormally high, then you might measure it again.
  • If the second measurement indicates that blood pressure is in the normal range, and you do not have proof that the first reading was a mistake, then do not record only the second

reading, either record both

measurements or the average

of the two readings.

scoring function mistakes
Scoring Function Mistakes
  • Mixing gains and losses
  • Not using scoring functions
  • Anchoring
scoring functions
Scoring functions
  • Objective value is translated to subjective worth
  • Input values become normalized output scores
  • Scoring functions must be elicited from the customer
percent happy scouts mistake
Percent happy scouts mistake
  • The Pinewood Derby tradeoff study had these criteria
    • Percent Happy Scouts
    • Number of Irate Parents
  • Because people evaluate losses and gains differently, the Preferred alternatives might have been different if they had used
    • Percent Unhappy Scouts
    • Number of Ecstatic Parents
slide47
Recommendation: Scoring functions in a tradeoff study

should express gains rather than losses.

not using scoring functions
Not using scoring functions
  • Most tradeoff studies that we have observed in industry did not use scoring functions.
  • In some cases, scoring functions were explained in the company’s engineering process, but they were not convenient, hence they were not used.
  • Recommendation: The Wymorian standard scoring functions should be used in tradeoff studies. Those located at http://www.sie.arizona.edu/sysengr/slides/,

should be referenced in company engineering processes.

anchoring
Anchoring
  • A person’s first impression dominates all further thought.
  • People were shown a wheel of fortune with numbers from one to hundred.
  • The wheel was spun and the subjects were asked to estimate the number of African nations in the United Nations.
  • If the wheel showed a small number, like 12, the subjects underestimated the correct number.
  • If the wheel showed a large number, like 92, the subjects overestimated the correct number.
  • Recommendation: When estimating values for parameters of scoring functions, think about the whole range of expected values for the parameters.
anchoring 2
Anchoring2
  • You should fill out a tradeoff study matrix row by row with the status quo as the first alternative. Therefore, the values of the status quo are the anchors for estimating the other data. Unfortunately, the status quo is likely to have extremely low values for performance and extremely high values for cost, schedule and risk. But at least the anchoring alternative is known, consistent and you have control over it.
  • Recommendations:
    • Make the status quo the first alternative.
    • In one iteration examine the scores left to right and in the next iteration examine them right to left.
output score mistakes
Output Score Mistakes
  • False precision
false precision
False precision
  • The most common mistake in tradeoff studies is false precision. For example, a tradeoff analyst asks an expert to estimate values for two criteria. The expert says, “The first criterion is about 2 and the second is around 3.” The analyst puts these numbers into a calculator and computes the ratio as 0.666666667. This is nonsense, but these nine significant digits are dragged throughout the tradeoff study. The Forer Effect might explain this: the analyst believes that the calculator is an impeccable authority in calculating numbers. Therefore, what the calculator says must be true.
  • Recommendation: In numerical tables, print only the number of digits after the decimal place that are necessary to show a difference between the preferred alternatives.
combining function mistakes
Combining Function Mistakes
  • Lack of knowledge
  • Lack of availability
lack of knowledge
Lack of knowledge
  • The average engineer is not familiar with the nuances of combining functions and their behavior specific to tradeoff studies.
  • Recommendation: Training with combining functions is necessary.
unavailability
Unavailability
  • Software is equipped with limited types of combining functions.
  • For example, one of the best commercial tools, Expert Choice, has only the Sum and the Product combining functions.
  • Most others have only the Sum.
  • Recommendation: Spreadsheet-formulated tradeoff studies have the greatest potential for combining function variety.
popular combining functions
Popular combining functions
  • Sum Combining Function = x + y
    • Used most often by engineers
  • Product Combining Function = x y
    • Cost to benefit ratio
    • Risk analyses
    • Game theory*
  • Sum Minus Product = x + y - xy
    • Probability theory
    • Fuzzy logic systems
    • Expert system certainty factors
  • Compromise =
preferred alternative mistakes
Preferred Alternative Mistakes
  • Overconfidence
  • Ignoring the need for expert opinion
  • Failure to talk with the customer
overconfidence
Overconfidence
  • Tradeoff studies are often started with over confidence.
  • The analyst prefers to maintain a state of over confidence without examining details.
  • Recommendation: For this bias, there is no better teacher than performing tradeoff studies, and bringing subjects to reviews and customer acceptance that require producing high-quality work in all tradeoff study components.
obviating expert opinion
Obviating expert opinion
  • The analyst holds a circular belief that expert opinion or review is not necessary because no evidence for the need of expert opinion is present.
  • Recommendation: Experts should be sought formally or informally to evaluate tradeoff study work.
  • In the last 40 years, the most common student mistake that Bahill has observed is failure to seek advice from experts and advisors.^
failure to talk with the customer
Failure to talk with the customer
  • The second most common mistake is failing to talk with the customer. Students and engineers seem to feel that it is a sign of weakness to ask for help.
  • Recommendation. Talk with your customer.
sensitivity analysis mistakes
Sensitivity Analysis Mistakes
  • Lack of training
  • Hawthorne effect
lack of training
Lack of training
  • Most engineers are not trained in sensitivity analyses.
  • Interactions among parameters can be very important.
  • Step sizes for the approximation of effects should be very small.
  • Second-order derivatives must be calculated accurately.
  • Recommendation: Investments in sensitivity analysis training must be made. Perhaps enabling software can substitute for much sensitivity analysis knowledge.
hawthorne effect
Hawthorne effect*
  • If you paint the factory walls a bright color, productivity goes up.
  • If you increase the illumination, productivity goes up.
  • If you decrease the illumination, productivity goes up.
  • Recommendation:
  • Maybe Hawthorne (and the Heisenberg Uncertainty Principle) affect evaluation data. If you measure something in a system, then its performance is likely to improve.
your job
Your job

is to help a decision maker make valid decisions that he or she (and other stakeholders) will have confidence in.

  • This is a difficult and iterative task.
  • It entails discovering the decision makers preferred weights, scoring functions, and combining functions.
  • You must also discover his or her mental mistakes and ameliorate them.
  • You must get into the head of the decision maker and discover his or her values*
personality types
Personality types
  • Different people have different personality types.
  • The Myers-Briggs model is one way of describing these personality types.
  • Sensory - Thinking – Judging people are likely to appreciate the tradeoff study techniques we have presented.
  • Intuitive – Feeling – Perceiving people most likely will not.
factors affecting human decisions
Factors affecting human decisions
  • the decision maker
    • corporate culture
    • the decision maker’s values
    • personality types
    • risk averseness
    • mental mistakes*
  • information display
    • wording of the question
    • context of presentation
  • the decision
    • effort required to make the decision
    • difficulty of making the decision
    • time allowed to make the decision
    • needed accuracy of the decision
    • cost of the decision
    • likelihood of regret
good industry practices
Good industry practices

for ensuring success of tradeoff studies include

  • having teams evaluate the data
  • evaluating the data in many iterations
  • expert review of the results and recommendations*
purpose of teaching tradeoff studies
Purpose of teaching tradeoff studies
  • Emotions, illusions, biases and use of heuristics make humans far from ideal decision makers.
  • Using tradeoff studies thoughtfully can help move your decisions from the normal human decision-making lower-right

quadrant to the ideal

decision-making

upper-left quadrant.

improving the tradeoff process
Improving the tradeoff process
  • Inform decision makers about how mental mistakes affect tradeoff studies, forewarned is forearmed
  • Creating a long-term, institutional decision horizon usually increases rationality
  • Team approach
  • Iterations
  • Public reviews
  • Reduce mental errors by using the recommendations given in this presentation
pr cis
Précis
  • Tradeoff studies seek to build a mathematical framework
  • The goal is a correct, parallel mathematical consideration of all relevant criteria, avoiding misjudgments associated with the serial consideration of criteria in subgroups
  • Cognitive science can help improve the validity and sensitivity of tradeoff studies
the rms titanic lifeboat decision
The RMS Titanic lifeboat decision

The original design for the RMS Titanic called for 64 lifeboats, but this was reduced to 20 before its maiden voyage: this might have been a mistake. The Chief Designer (CD) wanted 64 lifeboats. But the Program Manager (PM) reduced it to 20 after his advisors told him only 16 were required by law. The CD resigned over this decision. The British Board of Trade regulations of 1894 specified the lifeboat capacity. For ships over 10,000 tons, this lifeboat capacity was specified by volume (5,500 cubic feet), which could be converted into passenger seats (about 1000) or the number of lifeboats (about 16). So, even though the Titanic displaced 46,000 tons and was certified to carry 3,500 passengers, its 20 lifeboats complied with the regulations of the time. But let us go back to the design decision to reduce the number of lifeboats from 64 to 20. What if they had performed the following hypothetical tradeoff study? In this table, the weights of importance range from 0 to 10, with 10 being the most important and the evaluation data (scores) also range from 0 to 10, with 10 being the best. For simplicity, we have not used scoring functions, so the evaluation data are also the scores.

mental mistakes in the titanic decision 1
Mental mistakes in the Titanic decision1
  • The Program Manager and Chief Designer had different preferred alternatives because of their different weights of importance.
  • The PM had overconfidence in his subjective choice of 20 lifeboats. If he had done this tradeoff study, might he have rethought his decision?
  • In 1912, the White Star line said the Titanic was “unsinkable.” If the PM did not feel invincible, would he have authorized more lifeboats?
  • If the PM understood the Forer effect (that an analyst might fail to question or re-write criteria that originated from a perceived authority), might he have reassessed the Board of Trade’s requirement for 16 lifeboats?
mental mistakes in the titanic decision 2
Mental mistakes in the Titanic decision2
  • The PM and the CD did not do a tradeoff study.
  • They merely discussed the 20 and 64-lifeboat alternatives.
  • If they had understood distinctiveness by the addition of alternatives and had done this tradeoff study with the addition of the 10 and 30-lifeboat alternatives, would the PM have chosen a different alternative?
mental mistakes in the titanic decision 3
Mental mistakes in the Titanic decision3
  • A sensitivity analysis of the tradeoff study shows that the PM’s most important parameter is the weight of importance for the Cost criterion and that the CD’s most important parameter is the weight of importance for the Percentage of People that Could be Accommodated criterion.
  • Therefore, the PM should have spent more time assessing the magnitude and reliability of these weights.
  • In fact, he should have noted the importance of lifeboat regulations, and questioned whether such regulations were up-to-date for the new, larger Titanic design.
ad