Planning how to conduct the evaluation
Download
1 / 26

Planning How to Conduct the Evaluation - PowerPoint PPT Presentation


  • 283 Views
  • Updated On :

Planning How to Conduct the Evaluation. Dr. Suzan Ayers Western Michigan University (courtesy of Dr. Mary Schutten). Relationship of focus and planning. The structure of evaluation consists of: focusing the evaluation collecting, organizing, analyzing & reporting info

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Planning How to Conduct the Evaluation' - shelley


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Planning how to conduct the evaluation l.jpg

Planning How to Conductthe Evaluation

Dr. Suzan Ayers

Western Michigan University

(courtesy of Dr. Mary Schutten)


Relationship of focus and planning l.jpg
Relationship of focus and planning

  • The structure of evaluation consists of:

    • focusing the evaluation

    • collecting, organizing, analyzing & reporting info

    • administering the evaluation (see CIPP notes)

  • Evaluation should be conducted flexibly

  • Evaluator needs clear understanding of evaluation’s purpose and role


Slide3 l.jpg

  • Focusing the evaluation determines what information is needed

    • Origin and context of the proposed evaluation

    • Identifying/selecting questions, criteria, standards

  • Once the evaluation questions are known, the next step is to determine what information is needed to answer each question


Slide4 l.jpg

  • Planning the evaluation consists of collecting info, organizing info, analyzing info, reporting info, and administering the evaluation

    SAMPLE

  • Evaluation Question:Have the critical program activities occurred on time and within budget?

  • Things evaluator would need to know:

    • Which activities were viewed as critical?

    • Program time frames budget

    • Program time frames budget by activity

    • When each activity began/ended

    • Total cost of each critical activity


Identifying design and data collection methods l.jpg
Identifying Design andData Collection Methods

  • Involve client and stakeholders in deciding necessary info to best answer each question

  • Designs specify organization/structure for data collection

  • Causal designs: (quasi)experimental designs

    • Descriptive case studies, multiple regression or other stats methods to answer evaluation Qs

  • Descriptive designs: describe(case study), analyze the program, show a trend(time series), assess public opinions(cross-sectional), illustrate a process(thick description)

    • Commonly used in needs assessment and process studies


Slide6 l.jpg

  • Evaluator and stakeholders examine each question carefully to identify any important research design issues

  • Most evaluations are multiple research designs or combinations

  • Important to discuss early to see if:

    • groups are available, appropriateness of random assignment, time collecting data, etc…

    • Is the design “do-able?”


Identifying appropriate information sources l.jpg
Identifying Appropriate to identify any important research design issuesInformation Sources

  • Once needed information is agreed upon, the source(s) of that information must be specified

    • “Source”: group of individuals or location of existing information that answers each Q

    • Who will have information or access to it?

    • Who will be able to collect those data?


Slide8 l.jpg

  • Using existing data as information source to identify any important research design issues

    • Does necessary information already exist in a readily available form?

  • Commonly used information sources

    • Program recipients, deliverers, persons who have knowledge of the program recipients, public documents/databases

  • Policies that restrict information sources

    • Do policies exist concerning collecting data from clients or existing files?

      • Confidentiality, anonymity, privacy, IRB protocols


Slide9 l.jpg

  • Client involvement in identifying sources to identify any important research design issues

    • Evaluator, by training and experience, often can identify key sources of information

    • Client will be able to identify sources of information that may be missed by the evaluator

    • Collaboration yields helpful answers and enhances sense of shared ownership


Identifying data collection methods and instruments l.jpg
Identifying Data Collection Methods and Instruments to identify any important research design issues

  • Data collected directly from individuals identified as sources of information

    • Self reports

      • interviews, surveys, rating scales, focus groups, logs/journals

    • Personal Products:

      • tests (objective, essay), performances (simulations, role-play, competency testing), work samples (portfolios)

  • Data collected by independent observer

    • Narrative accounts

    • Observation forms (rating scales, checklists)


Slide11 l.jpg

  • Data collected with to identify any important research design issues technological device

    • Audiotape

    • Videotape

    • Time-lapse photos

    • Others

      • BP cuffs, speed, graphic recordings of performance skills, computer collation of participant responses

  • Data collection from unobtrusive measures


Slide12 l.jpg

  • Data collected from to identify any important research design issuesexisting information

    • Public documents

      • federal, state, local, databases etc..

    • Review of organizational documents

      • client files, notes of employees/directors, audits, minutes, publications

    • personal files

      • Correspondence, e-mails, etc.


Slide13 l.jpg

  • After identifying for use some of the methods on the next slides, it is important to review adequacy of techniques

    • Will the info collected provide a comprehensive picture?

    • Are the methods legal and ethical?

    • Is cost of data collection be worthwhile?

    • Can they be collected w/o undo disruption?

    • Can data be collected w/in time constraints?

    • Will the information be reliable and valid for the purposes of the evaluation?


Determining appropriate conditions for collecting information l.jpg
Determining Appropriate Conditions for Collecting Information

  • 3 key issues around data collection:

    • Will sampling be used?

    • How will data actually be collected?

    • When will data be collected?

  • Specifying sampling procedures to be employed

    • Sampling helps researcher draw inferences about the population in the study

    • Sampling is a tool to use whenever resources and time are limited

    • Sampling useful when it will not diminish the confidence of results

    • Sample size must be appropriate; too small a sample is of limited value


Slide15 l.jpg

  • Specifying how the information will be collected Information

    • Who will collect data?

    • For interviews, focus groups etc… how will characteristics of the evaluator influence data collection?

    • What training should be given to people collecting the data?

    • In what setting should data collection take place?

    • Confidentiality protected?

    • Special equipment, materials needed?

    • Evaluators need a bigger ‘bag of tools’ needed than most researchers to examine a broad array of phenomena


Slide16 l.jpg


Determining appropriate methods to organize analyze interpret information l.jpg
Determining Appropriate Methods to Organize, Analyze, Interpret Information

  • Develop a system to code, organize, store, and retrieve data

  • For each evaluation question, specify how collected information will be analyzed

    • Identify statistical/summarizing techniques

    • Designate some means for conducting the analysis

  • Interpreting results (Statistical reports do not speak for the themselves)

    • Share information with clients to gain perspective on potential interpretations of the data

    • Criteria/Standards guide interpretation of some Qs

    • Eval plan should allow recording of multiple or conflicting interpretations

    • Interpretations should consider multiple perspectives


Determining appropriate ways to report evaluation findings l.jpg
Determining Appropriate Ways to Interpret InformationReport Evaluation Findings

  • Using a matrix appropriate way to plan reporting (Fig 13.1)

    • Audience, content, format, date, context of presentation

  • Suggested Qs (Brknerhoff, Brethower, Hluchyj, & Nowakowski, 1983)

    • Are reporting audiences defined?

    • Are report formats and content appropriate for audience needs?

    • Will the evaluation report balanced information?

    • Will reports be timely and efficient?

    • Is the report plan responsive to the rights to info of the audiences?


Appropriate ways to report evaluation findings l.jpg
Appropriate Ways to Report Interpret InformationEvaluation Findings

Worksheet approach; summarize: 8 topics for each Q

1. Information required to answer the question

2. Design(s) to be used to collect information

3. Sources(s) of that information

4. Method(s) for information collection


Slide20 l.jpg

5. Information-collecting arrangements Interpret Information

-sampling procedure

-collection procedure

-schedule for collection

6. Analysis procedures

7. Interpretation procedures (including standards)

8. Reporting procedures

-audiences

-content

-format

-schedule

-context


The management plan l.jpg
The Management Plan Interpret Information

  • Final task in planning the evaluation study is describing how it will be carried out

  • Management plan essential to help oversee the project

    • Who will do it?

    • How much will it cost?

    • Will it be within budget?


Slide22 l.jpg

  • Evaluation management is multifaceted Interpret Information

    • supervise staff

    • serve as liaison to evaluation clients, participants, and stakeholders

    • identify and cope with political influences

    • communicate, communicate, communicate

  • Evaluation, whether by one person or a team, cannot afford to be disorganized or haphazard

  • Management plan needed to structure and control resources

  • Good management plans specify for ea eval Q (Fig 13.4) :

    • Tasks & timelines, personnel/resources for each task, cost


Slide23 l.jpg

  • Managing and estimating time for conducting evaluation Interpret Information

    • PERT (program eval & review technique) & Gantt charts are commonly used to estimate time on tasks

    • Gantt charts (Fig 13.6): simple displays that include chronologically scaled time frames for each evaluation task

      • Y (vertical) axis: tasks

      • X (horizontal) axis: time scale

      • Horizontal line drawn for each task show time needed

    • Help highlight interim deadlines or milestones that must be met to stay on the time in the study

    • Well-specified milestones essential monitoringtool


Slide24 l.jpg

  • Analyzing personnel needs and assignments Interpret Information

    • Quality of evaluation depends heavily on those who carry it out

    • Are qualified individuals available to carry out the tasks?

      • “Personnel role specifications” for all tasks (Suarez, 1981)

      • Specify who would manage the study, complete the eval design, select or develop instruments, collect data, analyze data, write summary reports, etc…


Slide25 l.jpg

  • Estimating costs and developing budgets Interpret Information

    • Staff salary and benefits

    • Consultants

    • Travel and per diem ( for staff/consultants)

    • Communications (postage, phone, etc.)

    • Printing and duplication

    • Data processing

    • Printed materials

    • Supplies and equipment

    • Subcontracts

    • Overhead (facilities, utilities)

      If initial estimate exceeds expectations, review each line item


Slide26 l.jpg

  • Agreements and contracts Interpret Information

  • Potential problems that arise during the evaluation can be more easily resolved if client and evaluator share a firm understanding

  • A well-documented agreement prior to launching the evaluation study concerning important procedures is very helpful

  • http://www.wmich.edu/evalctr/checklists/contracts.pdf


ad