project evaluation workshop n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Project Evaluation Workshop PowerPoint Presentation
Download Presentation
Project Evaluation Workshop

Loading in 2 Seconds...

play fullscreen
1 / 20

Project Evaluation Workshop - PowerPoint PPT Presentation


  • 70 Views
  • Uploaded on

Massachusetts Board of Higher Education Improving Teacher Quality Higher Education Partnership Program. Project Evaluation Workshop. Hoagland-Pincus Conference Center March 3, 2005. Workshop Objectives. Through an interactive process:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Project Evaluation Workshop' - nomlanga-harper


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
project evaluation workshop

Massachusetts Board of Higher Education

Improving Teacher Quality Higher Education Partnership Program

Project Evaluation Workshop

Hoagland-Pincus Conference Center

March 3, 2005

workshop objectives
Workshop Objectives
  • Through an interactive process:
  • Develop shared understanding of minimal expectations/standards for TQ project evaluations and reporting
  • Work towards a common conceptual model of professional development as basis for common approach to program evaluation
  • Identify next steps for technical assistance with evaluation plans
  • Support networking among projects to facilitate sharing of evaluation approaches, measures, tools
challenges
Challenges
  • Diversity among projects
    • Content areas
    • Target audiences
    • PD delivery mechanisms
    • Duration
    • Current stage of projects (new vs. ongoing)
    • etc., etc.
  • Resource limitations
agenda
Agenda
  • Introductions
  • BHE’s Evaluation and Reporting Expectations
  • State-level Data Collection
  • Logic Model of Professional Development
  • Planning for Project Evaluation
  • Next Steps
who we are background
Who We Are: Background
  • UMass Donahue Institute:
  • Public service and outreach arm of the UMass President’s Office
  • Broad range of services to federal, state, local public and non-profit organizations; services include applied social science research and program evaluation
  • Experience evaluating large statewide education reform initiatives for the Massachusetts Department of Education and Board of Higher Education
  • Experience developing and implementing evaluation plans for professional development and other educational programs and interventions at the local, regional, and state levels
who we are tq evaluation team
Who We Are: TQ Evaluation Team
  • Eric Heller, UMass Donahue Institute Director of Research and Evaluation

eheller@donahue.umassp.edu

413-587-2402

  • Christine Lewis, Research Manager

Lead Project Manager for TQ Evaluation

clewis@donahue.umassp.edu

413-587-2409

  • Jean Supel, Research Manager

Co-Project Manager for TQ Evaluation

jsupel@donahue.umassp.edu

508-856-1210

who we are functions of tq evaluation team
Who We Are: Functions of TQ Evaluation Team
  • Coordinate state-level collection of standardized project data and reports on behalf of Board of Higher Education
  • Provide technical assistance to projects in support of quality project evaluation efforts
  • Develop state level project report through aggregation of project data and meta-analysis of project reports
who you are introductions and project overview
Who You Are: Introductions and Project Overview
  • Subject Matter (e.g. Math, Language Arts, Math/Science, etc.)
  • Project Length
  • New or Existing Project
  • Target Population
    • Single or multiple cohort(s)
    • Teachers and/or Paraprofessionals; other
    • # of participants anticipated
    • School district(s) Involved
    • Grade level(s)
  • Type(s) of Professional Development Activities (e.g. Summer Institute, Graduate level course, After school group, classroom support, etc.)
  • Brief Description of Activities
evaluation reporting expectations 1
Evaluation & Reporting Expectations - 1
  • Develop and Implement a Project Evaluation that addresses:
    • Formative Evaluation Objectives
      • Provide timely feedback on project activities
      • Identify strengths and weaknesses
      • Identify gaps, unmet participant needs
      • Support continuous improvement of content and delivery
    • Summative Evaluation Objectives
      • Based on logic model of professional development
      • Document project implementation model (for replication)
      • Measure participation levels
      • Measure short-term participant outcomes (required)
      • Measure longer-term participant outcomes (to the extent feasible)
      • Measure student outcomes (to the extent feasible)
evaluation reporting expectations 2
Evaluation & Reporting Expectations - 2
  • Activity Tracking and Reporting
    • Target Audience
    • # Participants
    • Subject
    • Grade Level
    • Duration (# hours)
    • Timespan
    • # Credits

Note: ACTIVITY data for each completed PD activity will be collected annually through BHE online system

evaluation reporting expectations 3
Evaluation & Reporting Expectations - 3
  • School Tracking and Reporting
    • School Name
    • District
    • School Type (public/private/charter)
    • Poverty Level
    • # Participants by
      • Role (teacher/paraprofessional/administrator/other)
      • Grade level taught

Note: SCHOOL data for all participants will be collected annually through BHE online system

evaluation reporting expectations 4
Evaluation & Reporting Expectations - 4
  • Participant Tracking and Reporting
    • Basic standardized demographic and educational descriptive data to be collected from all participants in all activities. (See enclosed sample participant survey)
    • Form may be used as is, or incorporated into customized local survey
    • Electronic versions of survey to be distributed via email following workshop (PDF and Word)
    • Optional: Use of Participant Code to assist in participant tracking, linking to other local evaluation data

Note: PARTICIPANT data for every participant will be collected annually through BHE online system

evaluation reporting expectations 5
Evaluation & Reporting Expectations - 5

TQ Program Reporting Requirements

  • Annual report (multi-year projects) describing status, progress, milestones of project activities; includes evaluation progress report, with interim findings
  • Final report (all projects) summarizing project activities/milestones and final evaluation report
  • Evaluation reports (interim and final) to include project objectives, evaluation questions, methodology and results
  • General template for organizing evaluation report will be provided to facilitate state-level meta-analysis
  • Report to be submitted electronically
standard logic model of professional development
Standard Logic Model of Professional Development

Improved Student Outcomes

PD and Related Support Activities

Improved Instruction

Growth in Participant Skills / Knowledge

planning for project evaluation 1
Planning for Project Evaluation - 1

Logic Model Step 1 – PD and Related Support Activities

Sample Evaluation Questions:

  • To what extent have project activities been implemented as planned? What implementation challenges were encountered and how were they addressed?
  • Who participated and to what extent? In workshops? In follow up support activities?
  • To what extent were participant expectations/needs met?
  • How did participants perceive the quality of the activities?

Sample Data Sources:

  • Activity and participant tracking system
  • Staff interviews
  • Participant feedback – surveys, interviews, focus groups
planning for project evaluation 2
Planning for Project Evaluation - 2

Logic Model Step 2 – Growth in Participant Knowledge/Skills

Sample Evaluation Questions:

  • To what extent do participants achieve the stated learning objectives of PD activities?
  • To what extent do participants retain or deepen their understanding of concepts learned following PD? Do follow-up/support activities lead to enhanced understanding of concepts?
  • Do participants experience other benefits/outcomes (e.g., self-confidence?

Sample Data Sources:

  • Pre-test/post-test of PD content (short-term outcomes)
  • Follow-up administration of test (longer-term outcomes)
  • Other indicators of mastery – e.g., course assessments
  • Pre/post survey of attitudes, beliefs
planning for project evaluation 3
Planning for Project Evaluation - 3

Logic Model Step 3 – Improved Instruction

Sample Evaluation Questions:

  • To what extent do participants’ instructional practices change as a result of participation?
  • To what extent do participants incorporate material gained through PD into curriculum?
  • What challenges do participants encounter as they attempt to implement new skills, contents, approaches? How are challenges addressed?

Sample Data Sources:

  • Interviews, focus groups
  • Participant survey of self-reported changes
  • Classroom observation
planning for project evaluation 4
Planning for Project Evaluation - 4

Logic Model Step 4 – Improved Student Outcomes

Sample Evaluation Questions:

  • To what extent do students of participants achieve improved outcomes related to observed changes in instructional practices? Affective (attitudinal)? Cognitive?

Sample Data Sources:

  • Standardized assessments
  • Classroom assessments
  • Teacher perceptions (survey)
moving forward
Moving Forward
  • Our role is to work with you and your evaluator (internal or external) to implement an evaluation plan that:
  • Includes both formative and summative evaluation questions
  • Tracks activity, school, and participant data required for annual reporting
  • Has data collection and analysis organized around a basic logic model of professional development
  • Includes appropriate outcomes that measure targeted phases of the logic model
next steps
Next steps . . .
  • Projects complete evaluation plan summaries and submit to TQ Evaluation Team (today or via email within 1 week)
  • TQ Evaluation Team reviews evaluation plans
  • We will be in contact with you with feedback, questions, or possible suggestions – by phone, email, or on-site work session
  • Send us updated evaluation plan summaries as you further develop and/or revise you plan (email attachment)
  • Contact us with any questions or concerns