slide1
Download
Skip this Video
Download Presentation
Judah Leblang, Project Evaluator Program Evaluation and Research Group (PERG) at Lesley University

Loading in 2 Seconds...

play fullscreen
1 / 16

Judah Leblang, Project Evaluator Program Evaluation and Research Group (PERG) at Lesley University - PowerPoint PPT Presentation


  • 71 Views
  • Uploaded on

Coming from Behind: Developing a Logic Model and an Evaluation Strategy Late in a Center’s Life Cycle. Judah Leblang, Project Evaluator Program Evaluation and Research Group (PERG) at Lesley University. American Evaluation Association November 2, 2011. Evaluation of CELEST.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Judah Leblang, Project Evaluator Program Evaluation and Research Group (PERG) at Lesley University' - nonnie


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Coming from Behind: Developing a Logic Model and an Evaluation Strategy Late in a Center’s Life Cycle

Judah Leblang, Project Evaluator

Program Evaluation and Research Group (PERG) at Lesley University

American Evaluation Association

November 2, 2011

evaluation of celest
Evaluation of CELEST
  • PERG served as the primary evaluator for CELEST—The Center of Excellence in Learning in Education, Science and Technology in 2009-10.
  • The Center, based at Boston University, (also involving researchers from Brandeis, Harvard and MIT) was preparing for their critical review after receiving only provisional funding—an 18 month extension—in 2009.

Program Evaluation & Research Group (PERG) at Lesley University

what is celest
What is CELEST?
  • CELEST’s core scientific objective is to understand how the whole brain learns; i.e., how it adapts as an integrated system to enable intelligent autonomous behavior.
  • CELEST’s core technology objective is to develop novel brain-inspired technologies by implementing key insights gained from experiments and modeling: from bench to models to applications.

(CELEST website)

Program Evaluation & Research Group (PERG) at Lesley University

celest educational goals
CELEST: Educational Goals
  • CELEST is creating a new paradigm for educating graduate and undergraduate students in systems neuroscience by connecting biological knowledge about brains to an understanding of intelligent behavior through neural and computational models.
  • Accordingly, CELEST graduate students cross-train in experimental and modeling techniques in ways that will make them valuable members of cross-disciplinary teams of the future of systems neuroscience, because they will be fluent in more than one “language.”

(CELEST website)

Program Evaluation & Research Group (PERG) at Lesley University

celest evaluation priorities 2009 10
CELEST Evaluation Priorities 2009-10
  • Need for a comprehensive evaluation plan/credible evaluation
  • Need for a logic model and clarity around key goals
  • Demonstration of ‘value added’ or “centerness” as required by NSF
  • Integration of an external evaluator into the project planning process and data collection
  • Need to demonstrate change in leadership and restructuring of the project
  • Desire for external evaluator to oversee all aspects of the evaluation and respond to NSF/SLC requests

Program Evaluation & Research Group (PERG) at Lesley University

nsf slc evaluation priorities 2009 10
NSF/SLC Evaluation Priorities 2009-10
  • Rapid development of new/more comprehensive evaluation plan integrated into CELST project SIP
  • Data collection and analysis by January 2010
  • Examination of CELEST’s functioning as a center vs as a cluster of independent researchers
  • Measurements of “centerness,” to be compared with data from other SLC projects.
  • Development of Year 6 Evaluation Report as basis for site visit and critical review.

Program Evaluation & Research Group (PERG) at Lesley University

celest evaluation phase 1 sept 2009 january 2010
CELEST Evaluation: Phase 1 Sept 2009-January 2010
  • Documenting the value of CELEST as a center
    • Reviewed CELEST archival documents
    • Identified project strengths and examples of “centerness”
    • Conducted focus groups and interviews with graduate students and post-docs
    • Analyzed data and wrote evaluation report
    • Presented findings to CELEST board and site visit team—March 2010
  • Evaluation Methods used:
    • Two Focus groups: Interviewed 7 post docs, 6 graduate students
    • Student and Faculty survey data: Surveyed 28 of 34 grad students, 16 of 17 post docs; Surveyed 15 of 16 faculty
    • Interviews with PI and Co-PIs, 3 interviews with diversity team
    • Review of project website and archival documents

Program Evaluation & Research Group (PERG) at Lesley University

evaluation phase 2 fall 2009 ongoing
Evaluation Phase 2: Fall 2009--ongoing
  • Primary areas of focus:
      • “Centerness”:Value of CELEST as a center
        • Mechanisms to share resources and ideas about neuroscience and related technology, within centers, with partners, with broader audience—researchers, educators and general public
        • Document outgrowth of CELEST/benefits of collaborations
      • Education, outreach and partnerships
        • Education, research and career development
        • New Undergraduate courses--focus on CN 210—reviewed 2009 course materials and conferred with faculty and teaching fellows
        • Evaluate effectiveness of 2010 course
        • CELEST-related research experiences/internships
        • Partnerships with other institutions, industry, professional societies
      • Diversity
        • Documented CELEST activities promoting inclusiveness and diversity: WISE, SACNAS conferences, summer internships, other activities

Program Evaluation & Research Group (PERG) at Lesley University

slide9

Evaluation Questions

Data collection was driven by the following evaluation questions,

framed in consultation with CELEST staff, and developed to

satisfy SLC requirements:

What makes CELEST a center?

What does CELEST do that is different than research, educational, and other activities that would be carried out by a group of individual investigators?

What value [to the participating institutions, to students, to the field] does CELEST add and how?

Program Evaluation & Research Group (PERG) at Lesley University

evaluation questions cont d
Evaluation Questions cont’d

4. How, and how effectively does CELEST:

  • Facilitate interdisciplinary research collaborations among scientists and across institutions?
  • Support the education of a new interdisciplinary generation of scientists?
  • Increase the number of underrepresented group members, including women, people of various ethnic backgrounds, and people with disabilities entering the field?
  • Foster partnerships for the exchange of ideas with industry, other schools and professional societies?
  • Provide avenues for sharing resources and ideas within the center, with partners and with the larger community?

Program Evaluation & Research Group (PERG) at Lesley University

celest data collection methodologies
CELEST: Data Collection Methodologies

Program Evaluation & Research Group (PERG) at Lesley University

celest draft logic model fall 2009

Resources (includes NSF funding support)

Outputs &

measures

Short term outcomes—current year

Activities

Intermediate outcomes—18 months

Potential impact

CELEST Draft Logic Model—Fall 2009
celest draft logic model fall 2009 cont d

Resources (includes NSF funding support)

Outputs &

measures

Short term outcomes—current year

Intermediate outcomes—18 months

Activities

Potential impact

CELEST Draft Logic Model-Fall 2009 cont’d
key findings celest and centerness
Key Findings: CELEST and Centerness
  • More collaboration across departments at BU
  • More collaboration between BU and partner institutions (Harvard, MIT, Brandeis)
  • More collaboration across fields and areas of specialization
  • New funding structure: project, rather than investigator based
  • New projects established as outgrowths of CELEST (SYNAPSE/DARPA project)
  • CELEST research as “transformational”
  • Provides opportunity to do integrated research utilizing multiple approaches (modeling and experimental techniques)

Program Evaluation & Research Group (PERG) at Lesley University

celest key challenges
CELEST: Key Challenges
  • Uncertainty about funding and future of Center
  • Challenges related to transition to new management structure
  • Some resources still committed to projects from the previous configuration
  • Ongoing need to create and implement new systems

Program Evaluation & Research Group (PERG) at Lesley University

coming from behind lessons learned for evaluation
Coming from Behind: Lessons Learned for Evaluation
  • The importance of having a clear delineation of the evaluator’s role and responsibilities both within the project and between project staff and NSF;
  • The need for a flexible but detailed logic model, to serve as a visual representation of key project objectives and the means of achieving them;
  • The value of ongoing data collection by project staff and the need for leadership’s ‘buy in’ to the program evaluation process (ideally) from the project’s inception;
  • The need for clear/consistent guidelines from the funding agency, and provision of sufficient time and resources for a comprehensive evaluation to take place.

Program Evaluation & Research Group (PERG) at Lesley University

ad