Kristine L. Chadwick, Ph.D. Evaluator for The Spatial Intelligence and Learning Center (SILC)

1. Organizational Consultant, Critical Friend, and Evaluator: The Value and Challenge of Flexible Roles Kristine L. Chadwick, Ph.D. Evaluator for The Spatial Intelligence and Learning Center (SILC) The Science of Learning Center on Visual Language and Visual Learning (VL2)

2. SILC and VL2 Evaluations • NSF encouraged centers to change evaluators • Have served as SILC’s evaluator since its third year of operation (2008-2009) • Served as VL2’s evaluator during its fourth year of operation (2009-2010) (and will continue in Years 6-10)

3. What is SILC? (http://spatiallearning.org) “The Spatial Intelligence and Learning Center (SILC) brings together scientists and educators from Temple University, Northwestern University, the University of Chicago, the University of Pennsylvania, and the Chicago Public Schools (CPS) to pursue the overarching goals of • understanding spatial learning • using this knowledge to develop programs and technologies that will transform educational practice, helping learners to develop the skills required to compete in a global economy SILC participants include researchers from cognitive science, psychology, computer science, education, and neuroscience, as well as practicing geoscientists and engineers who are particularly interested in spatial thinking in their fields, and teachers in the CPS.”

4. What is VL2?(http://vl2.gallaudet.edu) “The purpose of VL2 is to gain a greater understanding of the biological, cognitive, linguistic, sociocultural, and pedagogical conditions that influence the acquisition of language and knowledge through the visual modality. Gallaudet University houses VL2, and brings together deaf and hearing researchers and educators from a variety of disciplines and institutions to study how language and literacy develop in deaf individuals.”

5. SILC and VL2 Evaluation Priorities • Satisfy NSF evaluation “requirements” • Present well at Year 4 high-stakes site visit • Consult on center management issues • Gather and report data on aspects of center PIs knew less about (e.g., outreach for SILC, training at affiliate IHEs for VL2)

6. Articulating Center Theory • Theory of the center, not its research per se • Value added of center-mode research • Challenging for the center scientists • Immediate and intermediate outcome categories • Centerness • Dissemination • Interdisciplinarity • Outreach • Training • Tools and resources

7. Program Theory / Logic Model • Long-term Outcomes (> 10 years) • Social Capital (reputation, networks) • Intellectual Capital (research advancement) • Physical Capital (tools and resources) • Human Capital (diverse interdisciplinary researchers) • Impacts based on NSF goals • advancing science • applying science to specific technological, educational, and workforce challenges • enabling science to capitalize on new opportunities and discoveries and respond to new challenges

8. Evaluation Approach • Operationalizing, measuring (or documenting) resources, activities, outputs, immediate outcomes in logic model • Reflect their growth as centers • Consulting on center processes • Outsider viewpoint, critical friend

9. NSF Requirements • Center-level evaluation requirements not very clear • Show value added of center-mode functioning • Present outcomes of center at site visits • NSF need: To demonstrate to the National Science Board the worth or value of the Science of Learning Centers

10. Limited Resource Tensions External Evaluator Consultant & Critical Friend Fix some process gaps Work through the logic model (are activities happening and resulting in expected outputs?) Attend to developmental stage of center Focus on process and some immediate outcomes • Show merit or worth of the center • Demonstrate how center-mode functioning better than individual RO1 grants • Focus on outcomes

11. Not enough resources to attend to all needs Caption for this vacation photo: “Up a Tree”

12. Conflicting Role Example • SILC Year 3 Evaluation • Process focused • Organizational consulting • Conversation and informal feedback and suggestions to Pis • SILC leadership satisfied • Site visit team wants outcome data

13. Example • SILC & VL2 Year 4 Evaluation • Outcomes oriented • Bibliometric analyses • Limited corpus of publications • Not useful to center at this early stage • Less feedback mid-year • PIs less satisfied • Fewer questions/critiques from site visit team

14. Balancing Tensions in a Most Unusual Way • Years 6-7 will bring back focus on process (new 5-year funding cycle) • Update logic model and key measures • Focus also on intermediate outcomes (now appropriate in center’s lifecycle) • “SLC Evaluation Network” economies of scale may result in better • center-specific evaluation • SLC program-wide evaluation

15. Thank You! Kristine Chadwick, Ph.D. Executive Director Evaluation and Applied Research Initiatives 304.347.0429 kristine.chadwick@edvantia.org