Dr. Ken Matwiczak Katherine Zackel Amanda Dillon Lyndon B. Johnson School of Public Affairs University of Texas at Aust

2. Genesis of the Project Fall 2008, US Advisory Commission on Public Diplomacy approached the LBJ School: Study current measurement methods for public diplomacy effectiveness Determine whether measurement is possible Create a catalogue of current PD efforts in sample countries Develop a quantitative and qualitative assessment model that will help identify most effective programs based on strategic goals

3. Original Timetable Sept to Mid-Jan: Background research Conduct survey and analyze results Coordinate with DoS Offices Develop plans to visit embassies Expand work on conceptual model Mid-Jan to July Draft measurement tool/model Conduct follow-up interviews, focus groups Get data from DoS to populate model Conduct training and get feedback at embassy locations Create and deliver report

4. Challenges and Limitations No funding until March No access to comprehensive information on PD programs or data No support or authority from DoS to survey PD/PA officers in the field No ability to train and get feedback from PD/PA officers in the field We were able to see some categories of data that was collected through the MAT, but not the data itself or all the categories.We were able to see some categories of data that was collected through the MAT, but not the data itself or all the categories.

5. Support Advisory Commission Staff and Commissioners Chereeka Montgomery, Director, EMU, Evaluation & Measurement Unit, OPP&R Rick Ruth, Director, ECA Office of Policy & Evaluation Robin Silver, Director of Evaluation, ECA Office of Policy and Evaluation Network of students� embassy contacts Chereeka Montgomery explained her process for gathering information and evaluating 8 missions Robin provided results from her studies and information on her gathering process Chereeka Montgomery explained her process for gathering information and evaluating 8 missions Robin provided results from her studies and information on her gathering process

6. Approach to the Project Interviewed guest experts Karen Hughes , Former Under Secretary of State for PD and PA Carl Chan, Executive Director of the Commission Gerald McLoughlin, Deputy Executive Director of the Commission David Firestein, Former Deputy Executive Director of the Commission Commissioners Peacock and Osborn Former Diplomats-in-Residence Greg Engle and Bill Stewart Surveyed PD professionals and academics Literature review and analysis of PD efforts Searched DoS and embassy websites

7. Analysis of Current Measurement Methods No single department coordinates measurement standards No unified effort among departments Duplication of evaluation efforts No uniform scale for comparing different programs Outcomes not clearly connected to programs Focus is on what happened, not what it means Not conducive for cost-effective analysis No formal relationship between DoS, EMU and PD about expectations, measurements, and results

8. Public Diplomacy � Model for Assessing Programs (PD-MAP) Developed three �desirable� outcomes of PD: Increasing understanding of US policy and culture Increasing favorability towards US Increasing the US influence in the world Created metrics for outcomes and sub-outcomes based on (anticipated) program strategic goals Designed a model to calculate metrics that could incorporate programs� strategic goals

9. PD-MAP Characteristics MS-Excel Spreadsheet Model Hierarchy of Performance Measures Expectations/Standards on a Common Scale Facilitates Strategic Planning & Communication Flexible & Adaptable

10. PD-MAP Input Elements Hierarchy of Performance Measures Outcomes (and Sub-Outcomes) Target Audiences Policy Area(s) of Interest Performance Metrics Priorities Measurement �Standards� �Ideal� Least Acceptable �Risk Characterization

11. Example Input (pg 1 - Hierarchy) Outcomes (1st level) Understanding Favorability Influence Audience (2nd level) Foreign Government Foreign �Elites� General Population

12. Example Input (pg 2 � Hierarchy cont�d) Policy Areas Culture Foreign Policy Economic Policy Security Policy Environmental Policy Performance Metrics % Change in Favorable Editorials in Foreign Press Change in number of anti-US Protests �etc.

13. Example Input (pg 3) Priorities (0-10 scale) Measurement Standard(s) (0=worst, 100=ideal) Risk Characterization Risk Averse Risk Taking Risk Neutral

14. PD-MAP Output(s) Performance �Scores� (0-100 scale) �Break-down� of Performance Scores Audit Trail of Results Graphical Display of Scores/Results

15. PD-MAP Uses Strategic Planning & Communication (at all levels) Ability to �Compare� Performance/Results Single Country/Program over Several Periods Multiple Countries/Programs over Single Period Identify Strong and Weak Results Sensitivity (�What If �) Analysis (based on historical results) Cost Effectiveness Assessment

16. Using PD-MAP Selecting Components Outcomes Audience (Sub-Outcomes) Policy Areas Programs Priority Metrics Performance Levels Risk

17. In Conclusion �Notional� Model Flexible, User-Friendly Program and/or Countries User-defined Elements Easily Understood Results Audit Trail �What-If� Analysis Facilitates Further Development Encourages (Requires) Strategic Planning & Communication

18. Questions & Discussion PD-MAP Public Diplomacy Model for Assessing Performance Contact: Dr. Ken Matwiczak LBJ School of Public Affairs University of Texas at Austin kmat@austin.utexas.edu (512) 471-7566