EVAL 6000: Foundations of Evaluation

EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2011

Agenda • Valuing theories and theorists • Semester review

Evaluation Theory Tree

Criticism and Connoisseurship • Grew out of methods used in art and literary criticism • Assumes that certain experts are capable of in-depth analysis and evaluation that could not be done in other ways • Based on evaluator’s special expertise and sensitivities • Methodologically, uses perceptual sensitivities, past experiences, refined insights, and ability to communicate assessment

Sensory evaluation • Flavor is comprised of • Retro nasal aroma • Basic taste • Chemical feeling factors • Aroma • Ortho nasal • Plug your nose • Eat a skittle with your nose plugged • Eat a second skittle • While chewing, unplug your nose

Consumer-Oriented Studies • The evaluator is the enlightened surrogate consumer • He or she must draw direct evaluative conclusions about the program being evaluated • Evaluation is viewed as the process of determining something’s merit, worth, and significance, with evaluations being the products of that process

Consumer-Oriented Studies • Regards consumer welfare as a program’s primary justification and accords that welfare the same primacy in program evaluation • Grounded in a deeply reasoned view of ethics and the common good, together with skills in obtaining and synthesizing pertinent, valid, and reliable information, the evaluator should help developers produce and deliver products and services that are of excellent quality and of great use to consumers

Consumer-Oriented Studies • Advance organizers include societal values, consumers’ needs, costs, and criteria of goodness in the particular evaluation domain • Purpose is to judge the relative merits, worth, or significance of the products and services of alternative programs and thereby to help taxpayers, practitioners, and potential recipients make wise choices

Consumer-Oriented Studies • Objectivist in assuming an underlying reality and positing that it is possible, although often extremely difficult, to find best answers • Often employ’s ‘goal-free evaluation’

Consumer-oriented comparison

Consumer-Oriented Studies • Employs a wide range of assessment methods • Largely, this approach applies Scriven’s general logic of evaluation in arriving at a summative statement • Establish criteria of merit • Construct standards • Measure performance and compare with standards • Synthesize and integrate into a judgment of merit or worth

Social Agenda and Advocacy Approaches • Aimed at increasing social justice through program evaluation • Seek to ensure that all segments of society have equal access to educational and social opportunities and services • Advocate affirmative action to give the disadvantaged preferential treatment through program evaluation • If, as many persons have stated, information is power, then these approaches employ program evaluation to empower the disenfranchised

Social Agenda and Advocacy Approaches • Favor a constructivist orientation and the use of qualitative methods (with the exception of transformative evaluation) • Eschew the possibility of finding right or best answers and reflect the philosophy of postmodernism, with its attendant stress on cultural pluralism, moral relativity, and multiple realities • Provide for democratic engagement of stakeholders in obtaining and interpreting findings

Responsive and Client-Centered Studies • Client-centered • Evaluator works with and for the support of a diverse client group • Continuous interaction, and response to, the evaluative needs of clients and stakeholders • Findings used primarily for improvement • Predominately applies case study methods

Constructivist Evaluation • Grounded in a rejection of the experimental philosophy—deeply paradigm driven • Sometimes known as 4th generation evaluation—because it goes beyond (1) objectives-based, (2) description, and (3) judgment • Intensive participation of stakeholders in the design, conduct, reporting, and application • Emphasis on the “constructions” (i.e., social realities) that different stakeholders bring to bear in assessing an evaluand

Constructivist Evaluation • Constructivist evaluation places the evaluators and program stakeholders at the center of the inquiry process, employing all of them as the evaluation’s “human instruments” • Constructivist evaluation is seen as a solution to problems in evaluations based on classical experimental design • These problems include nonuse of findings, objectification of human beings, the lack of meaningful involvement of stakeholders in evaluations, and the nonuse of evaluative processes by which people make sense of their worlds and those of others

Deliberative Democratic Evaluation • Functions within an explicit democratic framework • Charges evaluators to uphold democratic principles in reaching defensible conclusions • Three main features • Democratic participation • Dialogue to examine and authenticate stakeholder’s inputs • Deliberation to arrive at a defensible assessment of an evaluand’s merit and worth

Deliberative Democratic Evaluation • The approach has many advantages • It strives for democratic participation of stakeholders at all stages of the evaluation • It seeks to incorporate the views of all interested parties, including insiders and outsiders, disenfranchised persons and groups, as well as those who control the purse strings • Meaningful democratic involvement should direct the evaluation to the issues that people care about and make them inclined to respect and use the evaluation findings • The approach employs dialogue to examine and authenticate stakeholders’ inputs

Customer Feedback • The common practice by numerous commercial enterprises of obtaining and publishing on their Websites consumers’ ratings and (sometimes) narrative reviews of products or services they had supposedly purchased and used • In the marketing and communications literature, this approach is often referred to as ‘word of mouth’

Transformative Evaluation • The transformative evaluation approach represents a worldview, and its accompanying philosophical assumptions, that emerged from the writings of scholars from diverse ethnic and racial groups, people with disabilities, and feminists • Evaluators work in contested territory that is laden with pluralistic values and associated with real-life implications for the allocation of resources

Transformative Evaluation • From this point of view, knowledge claims are not neutral • They are influenced by human interest • All knowledge reflects power and social relationships within society • Transformative evaluation places a central importance on the lives of marginalized social groups (e.g., women, the poor, ethnic and racial minorities, people with disabilities)

Semester Review

Common Definition • Evaluation is the act or process of determining the merit, worth, or significance of something or the product of that process

Evaluative Predicates • Merit • Intrinsic quality; absent of context and costs • Worth • Synonymous with value; quality under consideration of costs and context • Significance • Synonymous with importance; merit and worth in a specific context

Evaluative Claims • Evaluators (ought to) make evaluative claims, or conclusions, beyond the typical empirical or research-based claims of “what’s so?” to inferences or evaluative claims of “so what?” • Evaluative claims of the “so what?” variety are couched in value-imbued language such as good or bad, priceless or worthless, and trivial or important, which are subsumed in the vocabularies of merit, worth, and significance

Competing Definitions • Evaluation is “the use of social science research procedures to systematically investigate the effectiveness of social intervention programs” (Rossi, Freeman, & Lipsey) • Proponents of theory-driven evaluation approaches characterize evaluation as explaining “how and why programs work, for whom, and under what conditions” (Chen)

“Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object” — William M. K. Trochim

Primary Purposes • Formative • Evaluation conducted with the intent to improve • Summative • Evaluation conducted with the intent to inform decision making and/or determine disposition

Program lifecycle and evaluation purpose

Secondary Purposes • Developmental • To help develop an intervention or program • Accountability • To hold accountable; usually under summative • Monitoring • To assess implementation and gauge progress toward a desired end • Ascriptive • Merely for the sake of knowing

Basic Functional Forms • Process evaluation • Assessment of everything that occurs prior to true outcomes • Outcome evaluation • Assessment of an evaluand’s effects • Cost evaluation • Assessment of monetary and non-monetary costs, direct and indirect costs, and actual and opportunity costs

Potential Direct Users • People who have authority over a program • People who have a direct responsibility for a program • People who are the intended beneficiaries of a program or who are impacted by a program (direct and indirect) • People disadvantaged by a program

Uses • Instrumental • Immediate decision making or action taking • Conceptual • Impact on the way people think about an evaluand • Process • Changes that occur as the result of an evaluation process • Persuasive • Decision makers use an evaluation to move others to support a decision

Uses and Misuses Use Misuse Ideal Use Mistaken Use (incompetence, uncritical acceptance, unawareness) Mischievous Use (manipulation, coercion) Instrumental Use Conceptual Use Persuasive Use Legitimate Use Misuse Rational Non-Use Political Non-Use Abuse (inappropriate suppression of findings) Unjustified Non-Use Justified Non-Use Non-Use

Politics of Evaluation • Evaluation is unavoidably implicated in matters of politics • Programs are created and maintained by political forces • Higher echelons of government, which make decisions about programs, are embedded in politics • The very act of evaluation has political connotations

Auditing versus Evaluation • Auditing is a procedure in which a third party systematically examines the evidence of adherence to some practice to a set of norms or standards for that practice and issues a professional opinion • This general idea spawned the notion of metaevaluation (Scriven)

Research versus Evaluation • The application of scientific logic and methods to social phenomena undertaken to advance knowledge or theoretical understanding (basic research) or with the intent of applying the results to some specific problem (applied research)

Evaluation Theory • Evaluation theories describe and prescribe what evaluators do or should do when conducting evaluations and are mostly normative in origin • They specify such things as evaluation purposes, users and uses, who participates in the evaluation process and to what extent, general activities or strategies, methods choices, and roles and responsibilities of the evaluator, among others

Logic of Evaluation • Scriven’s logic of evaluation is the closest to a meta-theory of evaluation • Establishing criteria • On what dimensions must the evaluand do well? • Constructing standards • How well should the evaluand perform? • Measuring performance and comparing with standards • How well did the evaluand perform? • Synthesizing and integrating information/data into a judgment of merit or worth • What is the merit or worth of the evaluand?

General Premises • Factual premises • The nature, performance, or impact of an evaluand or evaluee • Roughly equivalent to description (“what’s so?”) • Value premises • The properties or characteristics (i.e., criteria and standards) which typify a good, valuable, or important evaluand or evaluee of a particular class or type in a particular context

Value Premises • General values • The merit-defining criteria by which an evaluand or evaluee is evaluated; the properties or characteristics which define a ‘good’ or ‘valuable’ evaluand or evaluee • Specific values • The standards (i.e., levels of performance; usually an ordered set of categories) which are applied and by which performance is upheld, in order to determine if that performance is or is not meritous, valuable, or significant

EVAL 6000: Foundations of Evaluation