Behavioral Evidence that Measures the Valuation of Regional Water Quality Improvements

1. Behavioral Evidence that Measures the Valuation of Regional Water Quality Improvements Joel Huber--Duke University W. Kip Viscusi--Harvard University Jason Bell--Duke University From material originally presented at Valuation of Ecological Benefits Conference Wash D.C. October 2004

2. Introduction • Researchers on environmental benefits valuations have increased their use of the Internet, so that the performance of this survey approach has broad implications beyond our particular study. • The Knowledge Networks sample consists of a national probability sample of households recruited by random-digit dialing, who either have been provided internet access through their own computer or are given a WebTV device. • This replaces the mall intercept sample used in earlier versions of this work.

3. Agenda • Justification and description of the iterative paired choice method for valuation research • Sample representativeness • Statistical tests of the KN Panel • Implications for research & policy

4. Justification: Paired Choice Method (1) • The choice is between moving to an area with higher water quality vs. moving to one with a lower cost-of-living • Experimental conditions: • New area limits anchor on the current location • Unlike a referendum, the impact on others from the choice is minimal, thus minimizing altruism, envy or attitude towards government • Rather than model the distance of a person from a body of water, we model the percent of good water within a 100 mile radius

5. Sample Water Quality Valuation Choice

6. Justification: Paired Choice Method (2) • To test robustness of the results to different versions of the questionnaire, randomly identified groups received alternative versions. • These tests permit an assessment of the effects of anchoring and the initial range of the alternatives in the initial trade off.

7. Justification: Paired Choice Method (3) • For example, if the first choice is between a gain of 20% good in return for $400 in cost of living (e.g., $20 for one percentage point), then …. • respondents may reasonably use that information to assume that, $15 is a good price to pay for one percentage point gain. • By contrast, if the initial choice pits a 20% gain against $200, ($10 per one percentage point), then the $15 seems relatively high.

8. Justification: Paired Choice Method (4) • To test the impact of the initial range we altered the initial range in cost of living to be either $200, $300 or $500, • and the range of the gain in percent good to be either 20, 30 or 40 percentage points. • This test is whether the initial choice is appropriately sensitive to ranges, as required for appropriate sensitivity to scope.

9. Preparing Respondents for the Choices (1) • Respondents first think about their uses of lakes and rivers • Good lakes & rivers for a region 100 miles in radius are those good for fishing, swimming and aquatic environment • We define cost of living and ask how they would feel about a given gain in cost of living • Respondents make a practice choice where one option dominates on both cost of living and water quality to make sure they understand the concepts • All choices are relatively simple, involving just two options on two dimensions

10. Preparing Respondents for the Choices (2) • Each choice balances a benefit of better water quality against the value of lower cost of living • If the first choice favors water quality, the difference in water quality between the options is narrowed in the next question • If the first choice favors lower cost of living, the difference in cost of living between the options is narrowed • Choices progressively reveal the respondent’s value for good water

11. Results: Iterative Choice Approach • With over 2100 observations in five waves the appropriate value for 1% gain in water quality has a mean of about $27 and a median of $14 • These results are triangulated using choice-based conjoint analysis • The survey medium requires visual array of the choices in order for there to be respondent comprehension

12. Sample Representativeness • Comparison of the KN interviewed sample versus the Census demographic benchmarks

13. Sample Representativeness (1)

14. Sample Representativeness (2)

15. Sample Representativeness (3)

16. Summary of Sample Representativeness • The survey population closely mirrors the U.S. Census. • The hypothesis that people willing to be surveyed are better educated, extreme in income, or younger than the general population is not upheld. • While some differences are statistically significant they are not consequential. • 11 percent is age 64-74 while the national average is 9 percent • 21 percent have household incomes in the $50,000-$74,999 range while the national average is 19 percent. • “Overall, the sample tracks the U.S. population remarkably well.”

17. Statistical Tests of the KN Panel • Panel Tests: Effect of Panel Behaviors on Results • Interrupted interviews (break-off and resumed) • Panel respondent tenure • Attrition impact • Sample Self-Selection Bias Measurement

18. Panel Tests: Effect of Panel Behaviors (1) First: Did non-continuous response influence the valuation results? • These are the first such tests to have been undertaken for KN’s methodology. • Tested the panel influences via the regression analysis of the determinants of the value of water quality benefits. • First variable: respondents that stopped and then continued the survey at a later time. These may be less engaged in the survey task. • “There was no significant effect (-0.013) of this variable on benefit values.”

19. Panel Tests: Effect of Panel Behaviors (2) • Second: Does the time the respondent has been a member of the KN panel have an effect of these data? • Length of time on the panel may affect attentiveness to surveys or could be correlated with other personal characteristics that influence water quality valuations. • “The regression estimate (-0.0021) fails to indicate any significant influence of this variable.” Third: Do the number of days the respondent took to complete the survey reflect a lack of interest in the survey topic or in taking surveys generally? • “There is no significant effect (-0.006) of this variable on benefit valuations in either of the equations estimated.”

20. Panel Tests: Effect of Panel Behaviors (3) • Fourth: If the respondent quit the panel after the invitation for this survey, was there a deleterious effect on data quality? • Such respondents could be less interested in taking surveys and might have different valuations. The time gap was approximately between November 2002 to May 2004. • “However, this variable was also not statistically significant (-0.013) in the water quality valuation equations.”

21. Panel Tests: Summary of Effect of Panel Behaviors • “Overall, there is no indication that any of these key aspects of the panel methodology bias the survey responses.” • “In addition to the general match of our respondents to the U.S. population, we also examined whether these four variables reflecting the methodology had any influence on the probability that the respondent failed to pass the consistency test with respect to the benefit valuations.” • “There were no significant effects of any of the Knowledge Networks panel variables … there is no evidence that national performance of the survey task is importantly influenced by any of these variables.”

22. Sample Self-Selection Bias • Although the sample is nationally representative and had a high overall survey completion rate, it is useful to test for possible selection biases arising from panel members who did not successfully complete the survey. • Of 1,587 panel members invited to take the survey, 74% of respondents chose to participate. • Of the 1,174 participants, 3 respondents did not complete the portion of the survey that elicits water quality value. • Finally, 6% of participants completed the survey but were dropped because they chose the dominated alternative and continued with that choice even after being so informed. • Therefore, 1,103 of 1,587 invitees completed the water quality valuation portion of the survey.

23. Evaluating & Correcting for Selection Bias • Used variables for which there were values for non-respondents and respondents. (These data are routinely collected by KN and an advantage of this panel is that there is information from survey non-respondents.) • Predicted participation with variables that significantly affected survey completion, i.e. being African American or Hispanic which was negatively associated with completing the survey, as was household size. • Constructed two health-related stress dummy variables. • The first was for individuals that reported they had a high stress level as indicated by more “stress, strain, or pressure” than usual “during the past few months.” • The second was for people who failed to respond to the stress information question. • Each of these variables was negatively related to the probability of taking the survey, but not significantly related to the water quality valuation amount.

24. Effect of Sample Selection • The threshold empirical issue is whether there are any statistically significant selection effects. • The participation equation in the log unit regression analysis indicates that one can reject the hypothesis that there is a significant effect of sample selection on the estimates of valuation. • “Thus, the empirical estimates are not biased in any statistically significant way by the self-selection of respondents in the KN sample who chose to complete the survey and did so successfully.”

25. The survey results passed a variety of consistency and rationality checks • These included dominance tests as part of the iterative choice process and external scope tests across respondents. • The KN methodology itself was tested with respect to a variety of potential sources of bias, such as sample attrition, and these panel characteristics had no significant effect on the results.

26. Benefits of the KN Panel Approach • Warm-up questions involving dominated choices provided easy ways to understand the choice tasks. • The KN panel yielded a more representative sample than other survey methods such as those used by Magat in 2000 where a group of subjects contacted by phone had to come to a central location to take the survey. • KN respondents are accustomed to taking surveys, so they are not confused by the process. • The panel supported several statistical tests for assessment of non-response bias, including an estimate of the effect of self-selection bias on the survey results.

27. Practical Policy Application … • … provides unit water quality benefit values matched to existing EPA measures of water quality. • … demonstrates that the method provides stable, policy relevant estimates of the amount people are willing to pay for improvements in water quality; i.e. $23.17 for a one percentage point improvement. • … value is dependent on variables such as income, education , and personal use of lakes and rivers in the expected fashion. • … one can use these values to gauge the economic benefit of a local improvement project.

28. Implications for Researchers • Hypothetical market choice provides a measure of value that is less affected by altruism, guilt or attitudes towards government. • Paired choices are less influenced by external reference levels because they are direct one-to-one comparisons. • Paired choices are strongly affected by the starting values. • A survey method that allows respondents to see the relative choices produces cleaner data.

29. To obtain the complete paper … • This presentation updates a paper available at: • http://www.law.harvard.edu/programs/olin_center/papers/pdf/477.pdf