Evaluation Research

1. Evaluation Research • Describe how you might go about evaluating the effectiveness of a student/advisor relationship. • Discuss some of the potential political and ethical issues that might be involved in the study you described above. • Specify 3 social indicators that could be used in monitoring the effects of the Dean on the ‘career satisfaction’ of the faculty in the College of Education.

2. Evaluation Research

3. Evaluation research, sometimes called program evaluation, refers to a research purpose rather than a specific method. • This purpose is to evaluate the impact of social interventions such as new teaching methods, innovations in parole, and a host of others. • Evaluation research is a form of applied research—it is intended to have some real-world effect.

4. Many methods, like surveys and experiments can be used in evaluation research. • In recent years, the field of evaluation research has become an increasingly popular and active research specialty, as reflected in textbooks, courses, and projects.

5. Evaluation research reflects social scientists’ increasing desire to make a difference in the world. At the same time, there is the influence of • an increase in federal requirements that program evaluations must accompany the implementation of new programs, and • the availability of research funds to fulfill those requirements.

6. Topics Appropriate to Evaluation Research • Evaluation research is appropriate whenever some social intervention occurs or is planned. • Social intervention is an action taken within a social context for the purpose of producing some intended result. • In its simplest sense, evaluation research is the process of determining whether a social intervention has produced the intended result. • The topics appropriate for evaluation research are limitless. • The questions appropriate for evaluation research are of great practical significance: jobs, programs, and investments as well as values and beliefs.

7. Formulating the Problem: Issues of Measurement • Problem: What is the purpose of the intervention to be evaluated? • This question often produces vague results. • A common problem is measuring the “unmeasurable.” • Evaluation research is a matter of finding out whether something is there or not there, whether something happened or did not happen. • To conduct evaluation research, we must be able to operationalize, observe, and measure.

8. What is the outcome, or the response variable? • If a social program is intended to accomplish something, we must be able to measure that something. • It is essential to achieve agreements on definitions in advance. • In some cases you may find that the definitions of a problem and a sufficient solution are defined by law or by agency regulations; if so you must be aware of such specifications and accommodate them.

9. Whatever the agreed-upon definitions, you must also achieve agreement on how the measurements will be made. • There may be several outcome measures, for instance surveys of attitudes and behaviors, existing statistics, use of other resources.

10. Measuring Experimental Contexts • Measuring the dependent variable directly involved in the experimental program is only a beginning. • It is often appropriate and important to measure those aspects of the context of an experiment researchers think might affect the experiment. • For example, what is happening in the larger society beyond the experimental group, which may affect the experimental group.

11. Specifying Interventions • Besides making measurements relevant to the outcomes of a program, researchers must measure the program intervention—the experimental stimulus. • The experimental stimulus is the program intervention. • If the research design includes an experimental and a control group, then the experimental stimulus will be handled.

12. Assigning a person to the experimental group is the same as scoring that person “yes” on the stimulus, and assigning “no” to the person in the control group. • Considerations: who participates fully; who misses participation in the program periodically; who misses participation in the program a lot? • Measures may need to be included to measure level of participation. • The problems may be more difficult than that. • The factors to consider should be addressed thoroughly.

13. Specifying the Population • It is important to define the population of possible subjects for whom the program is appropriate. • Ideally, all or a sample of appropriate subjects will then be assigned to experimental and control groups as warranted by the study design. • Beyond defining the relevant population, the researcher should make fairly precise measurements of the variables considered in the definition.

14. New versus Existing Measures • If the study addresses something that’s never been measured before, the choice is easy—new measures. • If the study addresses something that others have tried to measure, the researcher will need to evaluate the relative worth of various existing measurement devices in terms of her or his specific research situations and purpose.

15. Advantages of creating measures: • They can offer greater relevance and validity than using existing measures. • Advantages of using existing measures: • Creating good measures takes time and energy, both of which could be saved by adopting an existing technique. • Of greater scientific significance, measures that have been used frequently by other researchers carry a body of possible comparisons that might be important to the current evaluation. • Finally, measures with a long history of use usually have known degrees of validity and reliability, but newly created measures will require pretesting or will be used with considerable uncertainty.

16. Operationalizing Success/Failure • Potentially one of the most taxing aspects of evaluation research is determining whether the program under review succeeded or failed. Definitions of “success” and “failure” can be rather difficult.

17. Cost-benefit analysis • How much does the program cost in relation to what it returns in benefits? • If the benefits outweigh the cost, keep the program going. • If the reverse, ‘junk it’. • Unfortunately this is not an appropriate analysis to make if thinking only in terms of money. • Ultimately, the criteria of success and failure are often a matter of agreement. • The people responsible for the program may commit themselves in advance to a particular outcome that will be regarded as an indication of success. • If that’s the case, all you need to do is make absolutely certain that the research design will measure the specified outcome.

18. Researchers must take measurement quite seriously in evaluation research, carefully determining all the variables to be measured and getting appropriate measures for each. • Such decisions are often not purely scientific ones. • Evaluation researchers often must work out their measurement strategy with the people responsible for the program being evaluated. • There is also a political aspect.

19. Types of Evaluation Research Designs • Evaluation research is not itself a method, but rather one application of social research methods. As such, it can involve any of several research designs. To be discussed: • 1.     Experimental designs • 2.     Quasi-experimental designs • 3. Qualitative evaluations

20. Experimental Designs • Many of the experimental designs introduced in Chapter 8 can be used in evaluation research.

21. Quasi-Experimental Designs: distinguished from “true” experiments primarily by the lack of random assignment of subjects to an experiments primarily by the lack of random assignment of subjects to an experimental and control group. In evaluation research, it’s often impossible to achieve such an assignment of subjects.

22. Rather than forgo evaluation all together, there are some other possibilities. • Time-Series Designs • Nonequivalent Control Groups • Multiple Time-Series Designs

23. Time-Series Designs: • Studies that involve measurements taken over time. See Figure 12-1 & 12-2. • 12-1 involves only an experimental group, without a control group.

24. Nonequivalent Control Groups: • Using an existing “control” group that appears similar to the experimental group, used when researchers cannot create experimental and control groups by random assignment from a common pool. • A nonequivalent control group can provide a point of comparison even though it is not formally a part of the study.

25. Multiple Time-Series Designs: • Using more than one time-series analysis. • These are the improved version of the nonequivalent control group design. • This method is not as good as the one in which control groups are randomly assigned, but it is an improvement over assessing the experimental group’s performance without any comparison. • See page 352, Figure 12-3

26. Qualitative Evaluations • Evaluations can be less structured and more qualitative. • Sometimes important, often unexpected information is yielded from in-depth interviews.

27. The Social Context • Evaluation research has a special propensity for running into problems. • Logistical problems • Ethical problems

28. Logistical Problems • Problems associated with getting subjects to do what they are supposed to do, getting research instruments distributed and returned, and other seemingly unchallenging tasks that can prove to be very challenging. • The special, logistical problems of evaluation research grow out of the fact that it occurs within the context of real life.

29. Although evaluation research is modeled after the experiment—which suggests that the researchers have control over what happens—it takes place within frequently uncontrollable daily life. • Lack of control can create real dilemmas for the researchers.

30. Administrative control: • The logistical details of an evaluation project often fall to program administrators. • What happens when the experimental stimulus changes in the middle of the experiment due to unforeseen problems (e.g. escaping convicts; inconsistency of attendance, or replacing original subjects with substitutes)? • Some of the data will reflect the original stimulus; other data will reflect the modification.

31. Ethical Issues • Ethics and evaluation are intertwined in many ways. • Sometimes the social interventions being evaluated raise ethical issues. They may involve political, ideological and ethical issues about the topic itself • Maybe the experimental program is of great value to those participating in it. • But what about the control group who is not receiving help? • For example, Tuskegee, Alabama study, page 356

32. Use of Research Results • Because the purpose of evaluation research is to determine the success or failure of social interventions, you might think it reasonable that a program would automatically be continued or terminated based on the results of the research. • It’s not that simple. • Other factors intrude on the assessment of evaluation research results, sometimes blatantly and sometimes subtly.

33. Three important reasons why the implications of the evaluation research results are not always put into practice. • The implications may not always be presented in a way that the nonresearchers can understand. • Evaluation results sometimes contradict deeply held beliefs • Vested interests in the programs underway

34. Social Indicators Research • Combining evaluation research with the analysis of existing data. • A rapidly growing field in social research involves the development and monitoring of social indicators, aggregated statistics that reflect the social condition of a society or social subgroup. • Researchers use indicators to monitor social life. • It’s possible to use social indicators data for comparison across groups either at one time or across some period of time. • Often doing both sheds the most light on the subject.

35. The use of social indicators is proceeding on two fronts: • Researchers are developing ever more-refined indicators; finding which indicators of a general variable are the most useful in monitoring social life • Research is being devoted to discovering the relationships among variables within whole societies

36. Computer Simulation: • As researchers begin compiling mathematical equations describing the relationships that link social variables to one another (for example, the relationship between growth in population and the number of automobiles), those equations can be stored and linked to one another in a computer. • With sufficient number of adequately accurate equations on tap, researchers one day will be able to test the implications of specific social changes by computer rather than in real life.

37. Evaluation research provides a means for us to learn right away whether a particular “tinkering” really makes things better. • Social indicators allow us to make that determination on a broad scale; coupling them with computer simulation opens up the possibility of knowing how much we would like a particular intervention without having to experience its risks.