Foundations of Evaluation: Insights from Chris Coryn and Lindsay Noakes

1. EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn & Lindsay A. Noakes October 1, 2009

2. Critical Reading Paper #1 • In 60 seconds • Which paper did you choose and why? • What is your major premise? • What evidence do you have to to support your premise?

3. Stage 1 Theories • Stage I theories can be characterized as a period of social problem solving • Emphasis was on scientific rigor and valid causal knowledge of the effects of social programs • Reflects the tenor of the 1960s, a period of rapid development in the social sciences • During this period many advances were made • Measurement theory (e.g., IRT) • Validity theory (e.g., internal, external, construct, and threats to them) • Experimentalism (e.g., experimental and quasi-experimental design)

4. Stage 1 Theories • Criterion 1: Theory of Social Programming • Programs largely viewed as immediately implementable solutions to social problems • Rested on assumptions that • Implementation and evaluation would be unambiguous • “Successes” would be adopted • A and B combined, would lead to ameliorating social problems • Concerned primarily with innovations/pilots/demonstrations that were manipulable

5. Stage 1 Theories • Criterion 2: Theory of Use • Dominant assumption was “naïve instrumentalism” • Effective programs would be maintained or scaled up and ineffective ones would be eliminated (i.e., a rationalist view) • Use would occur almost naturally • Evaluation findings, being scientifically rigorous, would inform decision making • Use was not a primary concern of Stage 1 theorists given the above assumptions

6. Stage 1 Theories • Criterion 3: Theory of Knowledge Construction • Priority was given to “truth” • The dominant view was that an external reality probably exists, though it may never be perfectly known • Valid knowledge could be constructed (more or less) • Emphasis on sources of bias and the best methods for reducing them

7. Stage 1 Theories • Criterion 4: Theory of Valuing • Theorists at this stage agree that valuable social programs solve important social problems • They disagree about how value claims are made and whose values should be given priority • Prescriptive versus descriptive valuing • This argument still prevails today with most modern theories ignoring theories of valuing (explicitly) • Only Scriven argues that a singular statement of value is the sine qua non of evaluation and the primary task of evaluators

8. Stage 1 Theories • Criterion 5: Theory of Practice • Again, emphasis on assessing programs effectiveness at solving important social problems • Primary task is summative claims or judgments • Distance is crucial to scientific rigor and integrity • Methods of arriving at summative statements were given priority • Predominately methods that provide unbiased estimates of effects (e.g., experimental)

9. Michael Scriven • Scriven is widely viewed a the first major evaluation theorist • Whether one agrees or disagrees with his views, he has influenced nearly every other theorist and theory • Cook claimed that “Not to know Scriven is to be illiterate in evaluation” • His contributions to the field include, but are not limited to… • Valuing theory • Formative and summative evaluation • Goal-free evaluation • Causal logic and cause-probing methods • The logic of evaluation • Metaevaluation

10. Donald Campbell • Like Scriven, Campbell has had a widespread impact on the field (and most other fields) • His greatest accomplishment is arguably the development and elaboration of experimental and quasi-experimental methods, taxonomies of validity, and threats to validity • Although never widely accepted, his “experimenting society” and “utopian idealism” also has influenced many fields and areas of thought

11. Contrasting Stage 1 Theories • Theories of Social Programming • Theories of Use • Theories of Knowledge Construction • Theories of Valuing • Theories of Practice

12. Further Discussion of the Readings • What specific questions were raised as you read the assigned readings for this week (or past weeks)? Is there something specific that you didn’t understand? • What problems remain unresolved regarding Stage 1 theories? • What other issues emerged from the readings that need further exploration?

13. Encyclopedia Entries for this Week • Bias • Causation • Checklists • Chelimsky, Eleanor • Conflict of interest • Countenance model of evaluation • Critical theory evaluation • Effectiveness • Efficiency • Empiricism • Independence • Evaluability assessment • Evaluation use • Fournier, Deborah • Positivism • Relativism • Responsive evaluation • Stake, Robert • Thick Description • Utilization of evaluation • Weiss, Carol • Wholey, Joseph