Relationships between Involvement and Use in the Context of Multi-site Evaluation

1. Relationships between Involvement and Use in the Context of Multi-site Evaluation American Evaluation Association Conference November 12, 2009

2. Beyond Evaluation Use • Four-year NSF grant to study the relationships between involvement in program evaluation and use/influence • Research team (2 co-PIs and 8 graduate students) based at the University of Minnesota • Context of four NSF-funded multi-site programs • Involvement and use by not directly intended (unintended) users

3. Framework for Involvement • Cousins and Whitmore’s (1998) Systematic Collaborative Inquiry • Control of the Evaluation • Stakeholder Selection • Depth of Participation • Burke’s (1998) Key Decision Points • Evaluation Stages • Activities • Levels of control

4. Framework for Use

5. Framework for Use and Influence

6. More Recent Developments • Kirkhart, 2000 • Evaluation Influence = capacity of persons or things to produce effects on others by intangible or indirect means (Kirkhart, 2000) • Map influence along three dimensions: source, intention, and time • Mark & Henry 2003, Henry & Mark 2004 • Intangible influence on individuals, programs, and communities • Focus on direct use of evaluation results or processes not adequate

7. “Beyond Evaluation Use” NSF Programs

8. Four Programs and their Evaluations • ATE: Advanced Technological Education—mainly community college level projects to enhance work force—evaluation included site visits, yearly survey • LSC: Local Systemic Change—professional development for STEM in K-12 school districts—evaluation included observations, interviews, and surveys

9. Four Programs and Their Evaluations (cont.) • CETP: Collaboratives for Excellence in Teacher Preparation—projects to improve STEM teacher education—evaluation included surveys and observations • MSP-RETA: Math Science Partnerships, Research Evaluation and Technical Assistance—evaluation technical assistance included national meetings and provision of consultants

10. Methods • Surveys of project PIs and evaluators in the four projects (645 respondents, 46%) • Document review • Interviews with key informant project personnel (29) • Citation analysis (246 documents 376 citations) • Survey of NSF PIs (191 respondents, 54.7%) • In-depth analytic case studies

11. Results • Perception of Evaluation Quality • Ability to conduct high quality evaluation • Be recognized as capable • Interface with NSF • Evaluators as brokers and negotiators • NSF leveraging involvement and use • Importance of dissemination • Life Cycles • Program • Projects • Individuals

12. Results • Project Control • Complete choice • Required involvement • Balance affects use • Community and Networking • Outreach • Development of a community of practice • Mutual respect • Skill sharing • Process use

13. Results • Tensions • Where best to spend time and money • Balance local and national evaluation • Balance project and evaluation goals • Uniqueness • Complex context • Individual responses

14. Implications • Participants differentially affected by the depth and breadth of involvement in evaluation activities. • Neither breadth nor depth was consistently predictive of perceived level of involvement . • Lack of consistency in perceived involvement and use makes measuring involvement challenging. • Any investigation likely to be substantially affected by the nature of the evaluation and the characteristics of the individual.

15. Limitations • Only four instances of large, multi-site NSF evaluations and therefore generalizations to other settings are not possible, although potentialities can be suggested. • The case studies themselves are based on self-report data along with some archival records. • The numbers of people surveyed and interviewed are small but appear to be at least representative of the groups included. • The instruments used for data gathering were developed as part of the project and therefore might not be valid as measures of involvement and use in other contexts.

16. Future Research • Research on the causal nature of involvement with evaluation use • Themes presented here provide fruitful areas for more investigation • Cross-case analysis provides a strong baseline for more positivistic research • Examine the issues raised here through quantitative path analytic procedures • Develop strong theories about the relationship between involvement and use that could form the basis for hypothesis formulation

17. Note This material is based upon work supported by the National Science Foundation under Grant No. REC 0438545. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

18. For Further Information Online - http://cehd.umn.edu/projects/beu/default.html E-mail – Lawrenz@umn.edu Research Team: • Dr. Frances Lawrenz • Dr. Jean A. King • Dr. Stacie Toal • Kelli Johnson • Denise Roseland