1 / 42

Trick or treatment? Evaluating the quality of structured risk management decisions

Trick or treatment? Evaluating the quality of structured risk management decisions. Joe Arvai The Ohio State University Decision Research. Outline. Structured DM in brief A practical example Questions of quality (3 experiments) Parting thoughts. SDM In Brief.

adia
Download Presentation

Trick or treatment? Evaluating the quality of structured risk management decisions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Trick or treatment?Evaluating the quality of structured risk management decisions Joe Arvai The Ohio State University Decision Research

  2. Outline Structured DM in brief A practical example Questions of quality (3 experiments) Parting thoughts

  3. SDM In Brief • Structured decision approaches have their roots in… • …the observation that people tend to have tremendous difficulty with making decisions that involve multiple objectives and, therefore, tradeoffs.

  4. SDM In Brief • Structured decision approaches have their roots in… • …studies of the constructive nature of preferences in response to available cues.

  5. SDM In Brief • Structured decision approaches are designed based on… • …“value focused thinking.”

  6. SDM In Brief • Structured decision approaches are designed based on… • …the literature dealing with normative decision making, specifically the steps required for a complete analysis of a given decision

  7. SDM In Brief • Structured decision approaches are designed based on… • …methods for decreasing the cognitive burden associated with complex choices.

  8. The Case of Water Use Planningin British Columbia • Work with B.C. Hydro on a comprehensive, stakeholder-based development of revised operating plans at all major hydroelectric facilities. • In response to increasing, competing demands on provincial water resources

  9. Scale of Study

  10. The Case of Water Use Planningin British Columbia • Multiple Objectives: • Electricity generation/trade • Environmental quality • Water, land, & air • Recreation opportunities • Cultural values • Learn over time; reduce uncertainty http://www.bchydro.com/wup/

  11. Water Use Planning in B.C.The Basic Steps • The approach is based on work with stakeholders on: • Eliciting objectives from various stakeholders. • Identifying a series of workable options for management. • Establishing attributes of/measures for each objective. • Generating a matrix across these objectives and options and addressing the tradeoffs that selecting one option over another entails.

  12. Water Use Planning in B.C.Participants • BC Hydro (Crown Corporation) • Federal Government • Provincial Government • Local Government • First Nations • Community Stakeholders • Home owners, business operators, etc.

  13. Water Use Planning in B.C.Objectives, Attributes, Measures

  14. Water Use Planning in B.C.e.g., Financial Objectives & Attributes

  15. Water Use Planning in B.C.e.g., RecreationObjectives & Attributes

  16. Water Use Planning in B.C.Assessing Value

  17. 1.0 0.8 0.6 0.4 0.2 RISK PROFILES Nat Sum Probability Range of Values Nat 60 40 100 Sum 80 10 140 140 20 60 100 Expected Low High Expected Value Water Use Planning in B.C.Risk & Uncertainty

  18. Water Use Planning in B.C.

  19. Water Use Planning in B.C.

  20. A High Quality Process? • Government Support • Approach originally conceived as a pilot project at a single site • Now implemented at 23 sites province wide (18 Water Use Plans completed to date) • Cost-effective • Original budget to complete all 23 plans = $25 Million • Revised budget = Approx. $14 Million • Participant Satisfaction • Historically adversarial groups work together • BC Hydro has a clear mandate • Consensus plans often achieved

  21. Experiment 1Quantitative Measures of Quality • Designed to compare two approaches for involving stakeholders in water use planning in B.C… • …small groups (7-10) • …both conditions provided with the same information • …“Structured” vs. “Unstructured” process

  22. UNSTRUCTURED 1. Self-rating questions 2. Technical Information newspaper article informative booklet audio-documentary 3. Group Discussion 4. Evaluation of policy alternatives referendum willingness to pay 5. Self-rating questions STRUCTURED 1. Self-rating questions 2. Technical Information newspaper article informative booklet audio-documentary 3. Decision structuring characterize values & objectives group discussion about values and objectives. objectives ranking/tradeoffs 4. Evaluation of policy alternatives referendum willingness to pay 5. Self-rating questions Experiment 1Quantitative Measures of Quality

  23. Experiment 1Results

  24. Experiment 1Results F

  25. Experiment 1Conclusions • Based on participants’ self-ratings and an analysis of deliberation periods, we conclude that the the structured approach leads to higher quality decisions. • Arvai, J. L., R. Gregory, and T. McDaniels. 2001. Testing a structured decision approach: Value-focused thinking for deliberative risk communication. Risk Analysis, 21: 1065-1076.

  26. Experiment 2Analyzing Choices • Decision structuring to alleviate embedding in environmental valuation • …when a good is assigned a higher value on its own vs. when it’s part of a more inclusive set. • e.g., 5:1 differences in WTP for a single vs. a set of disaster preparedness services

  27. Experiment 2Analyzing Choices • Providing much-needed structure may help to overcome the embedding problem by helping decision makers to think about the components of a valuation problem.

  28. VERSION A 1. Technical Information newspaper article informative booklet audio-documentary 2. Decision structuring characterize values & objectives group discussion about values and objectives. objectives ranking/tradeoffs 3. WTP 1 Rivers 4. WTP 10 River VERSION B 1. Technical Information newspaper article informative booklet audio-documentary 2. Decision structuring characterize values & objectives group discussion about values and objectives. objectives ranking/tradeoffs 3. WTP 10 Rivers 4. WTP 1 River Experiment 2Analyzing Choices

  29. Experiment 2Analyzing Choices No Embedding W/X = Y/Z 0.42  0.25 W > Y Z<X

  30. Experiment 2Conclusions • Based on participants’ mean WTP judgments, embedding was not alleviated (according to the ratio standard). • McDaniels, T., R. Gregory, J. L. Arvai, and R. Chuenpagdee. 2003. Decision structuring as a means of alleviating embedding in environmental valuation. Ecological Economics, 44: 33-46.

  31. Experiment 32-Part Evaluation • An experiment that would measure both self-ratings of quality and subjects’ choices…

  32. UNSTRUCTURED 1. Self-rating questions 2. Technical Information 3. Choice Task Funding allocations across three risk problems 4. Self-rating questions STRUCTURED 1. Self-rating questions 2. Technical Information 3. Decision Structuring Objectives ranking, tradeoffs Linking objectives with management problems 4. Choice Task Funding allocations across three risk problems 5. Self-rating questions Experiment 32-Part Evaluation

  33. Experiment 3Three Risk Problems

  34. Experiment 3Self-Rating Results

  35. 1=Overpopulation 2=Disease 3=Trails Experiment 3Ranked Objectives

  36. Experiment 3Funding Allocations

  37. Experiment 3Conclusions • Appears to be a disconnect between self-ratings of quality and actual funding choices • Wilson, R.S. and J. L. Arvai. 2004. Evaluating the quality of structured risk management decisions. In Review.

  38. Parting Thoughts • Level of facilitation, time for deliberation, increased attention to tradeoffs, etc. all seem to be critical.

  39. Parting Thoughts • Anecdotal observations or evaluations based on self-reports alone are likely insufficient for evaluating the quality of structured decision approaches.

  40. Parting Thoughts • Affective responses to stimuli exert powerful influences on risk judgments

  41. Parting Thoughts • Not suggesting that the outcomes of all structured decision making approaches are suspect.

  42. Thanks Tim McDaniels Robin Gregory Ying Chuenpagdee Robyn Wilson Louie Rivers Dan Ohlsen SSHRC NSF OARDC EPI

More Related