INTERPRETING AND COMMUNICATING CHECKOFF PROGRAM EVALUATION RESULTS Dr. Gary W. Williams

1. INTERPRETING AND COMMUNICATING CHECKOFF PROGRAM EVALUATION RESULTS Dr. Gary W. Williams Professor, Agricultural Economics Director, Texas Agribusiness Market Research Center (TAMRC) Dr. John P. Nichols Professor, Agricultural Economics Interim Head, Department of Agricultural Economics Texas A&M University College Station, Texas “Quantitative Evaluation of Advertising and Promotion Programs” Commodity Promotion Evaluation Workshop for Researchers and Managers 2006 Spring NEC-63 Meeting Monterey, California March 9, 2006

2. The Research Isn’t Done … … Until the Results Are Communicated!

3. Why Do Checkoff Groups Want Their Programs Evaluated? Simply to satisfy the legislative requirements To demonstrate their effectiveness in expanding markets and enhancing profit. To assist in the design and adjustment of the program’s long-run strategic plan To serve the information needs of contributors, industry, and other stakeholders To provide the necessary economic ammunition to respond to legal challenges

4. What Do Evaluation Studies Typically Provide? Measures of Program “Effectiveness” Retail Sales and Price Impacts Benefits to Producers and Other Program Contributors Other Measure of Effectiveness: - Production and Industry Growth Impacts - Market Share Impacts - Export Competitiveness Impacts

5. What Do Evaluation Studies Typically Provide? Aggregate Measures of Effectiveness Benefit-Cost Ratio (BCR): The total benefits of the program divided by the total costs of promotion. Marginal Rate of Return (MRR): The percent change in program benefits from a 1% change in advertising expenditures. Internal Rate of Return (IRR): The return that a firm earns if it expands or invests in itself rather than investing that money elsewhere.

6. Communication Dilemma Results of Checkoff Program Evaluations Are Often: Not Well Understood Considered “Unbelievable”

7. RESULTS NOT UNDERSTOOD Economic Jargon Often Gets in the Way “Advertising Elasticity” … instead of how responsive demand is to advertising “Marginal Effects of Advertising” … instead of the additional sales from each additional dollar in advertising “Spillover Effects of Advertising” … instead of how your advertising affects your competitors’ markets and how their advertising affects your market “Producer Surplus” “Economic Rent” … instead of more easily understood concepts like “profit” and “revenue”

8. - Retail BCR: increase in retail sales revenue per dollar invested in advertising - Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising - Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising - Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising - Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money RESULTS NOT UNDERSTOOD Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways

9. - Retail BCR: increase in retail sales revenue per dollar invested in advertising - Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising - Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising - Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising - Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money RESULTS NOT UNDERSTOOD Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways

10. RESULTS NOT UNDERSTOOD Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways - Retail BCR: increase in retail sales revenue per dollar invested in advertising - Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising - Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising - Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising - Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money

11. RESULTS NOT UNDERSTOOD Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways - Retail BCR: increase in retail sales revenue per dollar invested in advertising - Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising - Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising - Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising - Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money

12. RESULTS NOT UNDERSTOOD Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways - Retail BCR: increase in retail sales revenue per dollar invested in advertising - Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising - Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising - Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising -Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money

13. RESULTS NOT UNDERSTOOD Results Not Straight-Forward in Their Interpretation No Clear Criteria for Interpreting AMEs -BCR > 1 typically interpreted to mean that the returns to advertising > the costs -But how high must the BCR be in order to justify a conclusion that benefits of the program sufficiently exceed the costs to warrant continuation of the program? -Typical BCRs range from 2:1 to 10:1. So how do we interpret a BCR of 50:1 or 100:1? - And how do we interpret a BCR at the bottom of or below the typical range? Negative?

14. RESULTS NOT UNDERSTOOD Results Do Not Answer Fundamental Questions of Checkoff Contributors Is the Advertising a “Good” Investment for Those Who Pay for the Program? - Could they do better by keeping checkoff money and investing in other common investment alternatives? - Economists: A fundamental concern for program evaluations is the opportunity cost of the checkoff funds. - Non-Economists: If contributors think they could do better by investing checkoff funds in financial instruments and earn a greater return, then they will be unconvinced by an estimated BCR that is positive and “high.”

15. RESULTS NOT UNDERSTOOD Results Do Not Answer Fundamental Questions of Checkoff Contributors Whether or Not and How Contributors Personally Benefit from Their Contributions? - Contributors expect that the money they contribute to a Checkoff Program will be spent in such a way that they will individually be better off than they would have been without contributing. - The AMEs we typically provide do little to communicate how different groups of contributors with different economic characteristics benefit from checkoff program activities.

16. RESULTS NOT “BELIEVABLE” Costs Are Observable But Benefits Are Not Contributors See the Effects of Checkoff Assessments on their Bottom Lines But Have No Way of Knowing What Portion of Their Revenues Are Directly Attributable to Those Assessments - Evaluations specifically intended to measure the portion of industry revenues that can be attributed to checkoff promotion programs. - Compare actual sales or revenues to nebulous concepts like “what might have been earned in the absence of the program.” - We tell producers that the $5 or $6 of revenue earned for every $ spent on advertising is actually in their pockets but they just don’t know it.

17. RESULTS NOT “BELIEVABLE” Costs Are Observable But Benefits Are Not Contributors See the Effects of Checkoff Assessments on their Bottom Lines But Have No Way of Knowing What Portion of Their Revenues Are Directly Attributable to Those Assessments - Evaluations specifically intended to measure the portion of industry revenues that can be attributed to checkoff promotion programs. - Compare actual sales or revenues to nebulous concepts like “what might have been earned in the absence of the program.” This Concept Has Proved Difficult to Communicate. - We tell producers that the $5 or $6 of revenue earned for every $ spent on advertising is actually in their pockets but they just don’t know it.

18. RESULTS NOT “BELIEVABLE” Large BCRs Often Taken As Implying Large Market Impacts Checkoff Programs Are Only One of Many Factors That Impact Markets - Total checkoff assessment collections compared to farm sales are usually quite small (less than 0.5% in most cases). - Large BCRs can be calculated by dividing a small return by an even smaller cost. - So even if the BCR is large, the actual market impacts may actually be quite small and easily overwhelmed by the effects of the many other market forces. - Nevertheless, “large” positive BCRs are often taken to mean that the program has a large impact on market demand, prices, profitability.

19. RESULTS NOT “BELIEVABLE” Checkoff Boards Tend to Oversell Actual and Potential Program Impacts Contributors Come to Expect Large Impacts When the Impacts Are Usually Quite Modest - When they do not see the large effects they have been told to expect, they disbelieve the research results. - Contributors must be educated about the true potential of their checkoff programs. - Sell checkoff programs as tools to contribute to higher demand, prices, and profits in good years and to reduce downside pressure in bad years rather than as a panacea for the financial problems faced by contributors.

20. RESULTS NOT “BELIEVABLE” Focus on Inappropriate Measures of “Effectiveness” Market Price a Key Market Indicator But a Poor Indicator of Program Effectiveness - Price effects of advertising depend on how responsive supply is to price changes. - In industry with no supply controls and highly elastic (price responsive) supply, most benefits of advertising come from a sales increase rather than from a price increase. - In industry with inelastic (price unresponsive) supply, most advertising benefits come from price increase rather than from sales increase. - BUT long-run supply response can dampen price effects of advertising over many years.

21. RESULTS NOT “BELIEVABLE” Focus on Inappropriate Measures of “Effectiveness” The Appropriate Measure of “Effectiveness” Depends on Objectives of Promotion Program - We tend to report BCRs and other AMEs because we assume that the driving objective of all promotion programs is to enhance profitability. - Strategic Plans often reveal other objectives or sub-objectives of more immediate interest and relevance.

22. 1. Increase American Lamb demand by 19% by 2010 2. Protect & enhance the business climate for American lamb 3. Leverage the ALB budget 4. Expand markets and market share for American lamb 5. Establish open and interactive communications with all segments of the industry 6. Gather research data and evaluate programs ALB VISION “A unified, thriving American Lamb Industry that concentrates its resources around a prioritized and measurable plan that fosters the opportunity for profitability of all Contributors.” ALB STRATEGIC PLAN

23. 1. Increase American Lamb demand by 10% by 2010 2. Protect & enhance the business climate for American lamb 3. Leverage the ALB budget 4. Expand markets and market share for American lamb 5. Establish open and interactive communications with all segments of the industry 6. Gather research data and evaluate programs ALB VISION “A unified, thriving American Lamb Industry that concentrates its resources around a prioritized and measurable plan that fosters the opportunity for profitability of all Contributors.” ALB STRATEGIC PLAN

24. IMPROVING COMMUNICATION Recognize that Communication is a Two-Way Street For Researchers to Communicate Better, Then Checkoff Groups Need to Better Communicate What They Want from an Evaluation - What do you want to know about your programs? - What metrics do you focus on? - What implies program success to you? Increased Sales? Higher Price? Industry Growth? Reduced Imports? Increased Exports?

25. IMPROVING COMMUNICATION Recognize that Communication is a Two-Way Street Also, Researchers Need to Know Why a Program Evaluation is Wanted - Legislative reporting requirements? Let the researchers design the study objectives and measures of effectiveness. - Assist in program management? Researchers need to know what that means to you so they can provide measures to meet your needs. - Strategic Plans provide insights into appropriate evaluation objectives and measures.

26. IMPROVING COMMUNICATION Inform Boards About What Can and Cannot Be Done They Usually Do Not Understand Extensive Data Requirements They Need to Understand the Types of Evaluations Typically Not Done By Agricultural Economists - Consumer attitude and awareness studies typically done by advertising firms, market research firms, and business school researchers. - The typical checkoff program evaluation attempts to measure directly the effects of advertising on demand without concern for attitudes and awareness. RETAIL SALES/PRICE PROMOTION

27. IMPROVING COMMUNICATION Inform Boards About What Can and Cannot Be Done They Also Need to Understand What Our Tools Will Allow US to Do and NOT Do Effectively Provide reasonable estimates of the returns to checkoff program activities in the aggregate. - CAN DO EFFECTIVELY: Provide reasonable estimates of the returns to specific checkoff program activities. - CANNOT DO EFFECTIVELY: ? Firm Performance

28. IMPROVING COMMUNICATION Inform Boards About What Can and Cannot Be Done They Also Need to Understand What Our Tools Will Allow US to Do and NOT Do Effectively - CAN DO EFFECTIVELY: Provide reasonable estimates of the returns to checkoff program activities in the aggregate. - CANNOT DO EFFECTIVELY: Provide reasonable estimates of the returns to specific checkoff program activities. All Firm Activities Firm Performance

29. IMPROVING COMMUNICATION Other Ways to Improve Communication Reduce economic jargon in reports Standardize and categorize measures of program effectiveness Develop measures that more readily convey the success or failure of checkoff programs in meeting their objectives Be careful not to imply that large estimated returns mean large program effects on demand, prices, profits, etc. Inform and educate on what checkoff programs can reasonably be expected to achieve in terms of magnitudes of effect

30. $ Sales $ Advertising Expenditures IMPROVING COMMUNICATION Other Ways to Improve Communication Be careful not to oversell what advertising programs can accomplish If you take all the credit for periods when sales and prices are growing … … then you’ll have to take the blame if sales and price ever drop. And contributors won’t believe the estimates of positive returns to the advertising paid for with their checkoff assessments.