Eval 6000 foundations of evaluation
This presentation is the property of its rightful owner.
Sponsored Links
1 / 41

EVAL 6000: Foundations of Evaluation PowerPoint PPT Presentation


  • 73 Views
  • Uploaded on
  • Presentation posted in: General

EVAL 6000: Foundations of Evaluation. Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2012. Agenda. Valuing theories and theorists Semester review. Evaluation Theory Tree. Criticism and Connoisseurship. Grew out of methods used in art and literary criticism

Download Presentation

EVAL 6000: Foundations of Evaluation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Eval 6000 foundations of evaluation

EVAL 6000: Foundations of Evaluation

Dr. Chris L. S. Coryn

Kristin A. Hobson

Fall 2012


Agenda

Agenda

  • Valuing theories and theorists

  • Semester review


Evaluation theory tree

Evaluation Theory Tree


Criticism and connoisseurship

Criticism and Connoisseurship

  • Grew out of methods used in art and literary criticism

  • Assumes that certain experts are capable of in-depth analysis and evaluation that could not be done in other ways

  • Based on evaluator’s special expertise and sensitivities

  • Methodologically, uses perceptual sensitivities, past experiences, refined insights, and ability to communicate assessment


Sensory evaluation

Sensory evaluation

  • Flavor is comprised of

    • Retro nasal aroma

    • Basic taste

    • Chemical feeling factors

  • Aroma

  • Ortho nasal

  • Plug your nose

  • Eat a skittle with your nose plugged

  • Eat a second skittle

  • While chewing, unplug your nose


Consumer oriented studies

Consumer-Oriented Studies

  • The evaluator is the enlightened surrogate consumer

  • He or she must draw direct evaluative conclusions about the program being evaluated

  • Evaluation is viewed as the process of determining something’s merit, worth, and significance, with evaluations being the products of that process


Consumer oriented studies1

Consumer-Oriented Studies

  • Regards consumer welfare as a program’s primary justification and accords that welfare the same primacy in program evaluation

  • Grounded in a deeply reasoned view of ethics and the common good, together with skills in obtaining and synthesizing pertinent, valid, and reliable information, the evaluator should help developers produce and deliver products and services that are of excellent quality and of great use to consumers


Consumer oriented studies2

Consumer-Oriented Studies

  • Advance organizers include societal values, consumers’ needs, costs, and criteria of goodness in the particular evaluation domain

  • Purpose is to judge the relative merits, worth, or significance of the products and services of alternative programs and thereby to help taxpayers, practitioners, and potential recipients make wise choices


Consumer oriented studies3

Consumer-Oriented Studies

  • Objectivist in assuming an underlying reality and positing that it is possible, although often extremely difficult, to find best answers

  • Often employ’s ‘goal-free evaluation’


Consumer oriented comparison

Consumer-oriented comparison


Consumer oriented studies4

Consumer-Oriented Studies

  • Employs a wide range of assessment methods

  • Largely, this approach applies Scriven’s general logic of evaluation in arriving at a summative statement

    • Establish criteria of merit

    • Construct standards

    • Measure performance and compare with standards

    • Synthesize and integrate into a judgment of merit or worth


Social agenda and advocacy approaches

Social Agenda and Advocacy Approaches

  • Aimed at increasing social justice through program evaluation

  • Seek to ensure that all segments of society have equal access to educational and social opportunities and services

  • Advocate affirmative action to give the disadvantaged preferential treatment through program evaluation

  • If, as many persons have stated, information is power, then these approaches employ program evaluation to empower the disenfranchised


Social agenda and advocacy approaches1

Social Agenda and Advocacy Approaches

  • Favor a constructivist orientation and the use of qualitative methods (with the exception of transformative evaluation)

  • Eschew the possibility of finding right or best answers and reflect the philosophy of postmodernism, with its attendant stress on cultural pluralism, moral relativity, and multiple realities

  • Provide for democratic engagement of stakeholders in obtaining and interpreting findings


Responsive and client centered studies

Responsive and Client-Centered Studies

  • Client-centered

  • Evaluator works with and for the support of a diverse client group

  • Continuous interaction, and response to, the evaluative needs of clients and stakeholders

  • Findings used primarily for improvement

  • Predominately applies case study methods


Constructivist evaluation

Constructivist Evaluation

  • Grounded in a rejection of the experimental philosophy—deeply paradigm driven

  • Sometimes known as 4th generation evaluation—because it goes beyond (1) objectives-based, (2) description, and (3) judgment

  • Intensive participation of stakeholders in the design, conduct, reporting, and application

  • Emphasis on the “constructions” (i.e., social realities) that different stakeholders bring to bear in assessing an evaluand


Constructivist evaluation1

Constructivist Evaluation

  • Constructivist evaluation places the evaluators and program stakeholders at the center of the inquiry process, employing all of them as the evaluation’s “human instruments”

  • Constructivist evaluation is seen as a solution to problems in evaluations based on classical experimental design

    • These problems include nonuse of findings, objectification of human beings, the lack of meaningful involvement of stakeholders in evaluations, and the nonuse of evaluative processes by which people make sense of their worlds and those of others


Deliberative democratic evaluation

Deliberative Democratic Evaluation

  • Functions within an explicit democratic framework

  • Charges evaluators to uphold democratic principles in reaching defensible conclusions

  • Three main features

    • Democratic participation

    • Dialogue to examine and authenticate stakeholder’s inputs

    • Deliberation to arrive at a defensible assessment of an evaluand’s merit and worth


Deliberative democratic evaluation1

Deliberative Democratic Evaluation

  • The approach has many advantages

    • It strives for democratic participation of stakeholders at all stages of the evaluation

    • It seeks to incorporate the views of all interested parties, including insiders and outsiders, disenfranchised persons and groups, as well as those who control the purse strings

    • Meaningful democratic involvement should direct the evaluation to the issues that people care about and make them inclined to respect and use the evaluation findings

    • The approach employs dialogue to examine and authenticate stakeholders’ inputs


Customer feedback

Customer Feedback

  • The common practice by numerous commercial enterprises of obtaining and publishing on their Websites consumers’ ratings and (sometimes) narrative reviews of products or services they had supposedly purchased and used

  • In the marketing and communications literature, this approach is often referred to as ‘word of mouth’


Transformative evaluation

Transformative Evaluation

  • The transformative evaluation approach represents a worldview, and its accompanying philosophical assumptions, that emerged from the writings of scholars from diverse ethnic and racial groups, people with disabilities, and feminists

  • Evaluators work in contested territory that is laden with pluralistic values and associated with real-life implications for the allocation of resources


Transformative evaluation1

Transformative Evaluation

  • From this point of view, knowledge claims are not neutral

    • They are influenced by human interest

    • All knowledge reflects power and social relationships within society

  • Transformative evaluation places a central importance on the lives of marginalized social groups (e.g., women, the poor, ethnic and racial minorities, people with disabilities)


Semester review

Semester Review


Common definition

Common Definition

  • Evaluation is the act or process of determining the merit, worth, or significance of something or the product of that process


Evaluative predicates

Evaluative Predicates

  • Merit

    • Intrinsic quality; absent of context and costs

  • Worth

    • Synonymous with value; quality under consideration of costs and context

  • Significance

    • Synonymous with importance; merit and worth in a specific context


Evaluative claims

Evaluative Claims

  • Evaluators (ought to) make evaluative claims, or conclusions, beyond the typical empirical or research-based claims of “what’s so?” to inferences or evaluative claims of “so what?”

  • Evaluative claims of the “so what?” variety are couched in value-imbued language such as good or bad, priceless or worthless, and trivial or important, which are subsumed in the vocabularies of merit, worth, and significance


Competing definitions

Competing Definitions

  • Evaluation is “the use of social science research procedures to systematically investigate the effectiveness of social intervention programs” (Rossi, Freeman, & Lipsey)

  • Proponents of theory-driven evaluation approaches characterize evaluation as explaining “how and why programs work, for whom, and under what conditions” (Chen)


Eval 6000 foundations of evaluation

“Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object”

— William M. K. Trochim


Primary purposes

Primary Purposes

  • Formative

    • Evaluation conducted with the intent to improve

  • Summative

    • Evaluation conducted with the intent to inform decision making and/or determine disposition


Eval 6000 foundations of evaluation

Program lifecycle and evaluation purpose


Secondary purposes

Secondary Purposes

  • Developmental

    • To help develop an intervention or program

  • Accountability

    • To hold accountable; usually under summative

  • Monitoring

    • To assess implementation and gauge progress toward a desired end

  • Ascriptive

    • Merely for the sake of knowing


Basic functional forms

Basic Functional Forms

  • Process evaluation

    • Assessment of everything that occurs prior to true outcomes

  • Outcome evaluation

    • Assessment of an evaluand’s effects

  • Cost evaluation

    • Assessment of monetary and non-monetary costs, direct and indirect costs, and actual and opportunity costs


Potential direct users

Potential Direct Users

  • People who have authority over a program

  • People who have a direct responsibility for a program

  • People who are the intended beneficiaries of a program or who are impacted by a program (direct and indirect)

  • People disadvantaged by a program


Eval 6000 foundations of evaluation

Uses

  • Instrumental

    • Immediate decision making or action taking

  • Conceptual

    • Impact on the way people think about an evaluand

  • Process

    • Changes that occur as the result of an evaluation process

  • Persuasive

    • Decision makers use an evaluation to move others to support a decision


Uses and misuses

Uses and Misuses

Use

Misuse

Ideal Use

Mistaken Use

(incompetence, uncritical acceptance, unawareness)

Mischievous Use

(manipulation, coercion)

Instrumental Use

Conceptual Use

Persuasive Use

Legitimate Use

Misuse

Rational Non-Use

Political Non-Use

Abuse

(inappropriate suppression of findings)

Unjustified Non-Use

Justified Non-Use

Non-Use


Politics of evaluation

Politics of Evaluation

  • Evaluation is unavoidably implicated in matters of politics

    • Programs are created and maintained by political forces

    • Higher echelons of government, which make decisions about programs, are embedded in politics

    • The very act of evaluation has political connotations


Auditing versus evaluation

Auditing versus Evaluation

  • Auditing is a procedure in which a third party systematically examines the evidence of adherence to some practice to a set of norms or standards for that practice and issues a professional opinion

  • This general idea spawned the notion of metaevaluation (Scriven)


Research versus evaluation

Research versus Evaluation

  • The application of scientific logic and methods to social phenomena undertaken to advance knowledge or theoretical understanding (basic research) or with the intent of applying the results to some specific problem (applied research)


Evaluation theory

Evaluation Theory

  • Evaluation theories describe and prescribe what evaluators do or should do when conducting evaluations and are mostly normative in origin

  • They specify such things as evaluation purposes, users and uses, who participates in the evaluation process and to what extent, general activities or strategies, methods choices, and roles and responsibilities of the evaluator, among others


Logic of evaluation

Logic of Evaluation

  • Scriven’s logic of evaluation is the closest to a meta-theory of evaluation

    • Establishing criteria

      • On what dimensions must the evaluand do well?

    • Constructing standards

      • How well should the evaluand perform?

    • Measuring performance and comparing with standards

      • How well did the evaluand perform?

    • Synthesizing and integrating information/data into a judgment of merit or worth

      • What is the merit or worth of the evaluand?


General premises

General Premises

  • Factual premises

    • The nature, performance, or impact of an evaluand or evaluee

    • Roughly equivalent to description (“what’s so?”)

  • Value premises

    • The properties or characteristics (i.e., criteria and standards) which typify a good, valuable, or important evaluand or evaluee of a particular class or type in a particular context


Value premises

Value Premises

  • General values

    • The merit-defining criteria by which an evaluand or evaluee is evaluated; the properties or characteristics which define a ‘good’ or ‘valuable’ evaluand or evaluee

  • Specific values

    • The standards (i.e., levels of performance; usually an ordered set of categories) which are applied and by which performance is upheld, in order to determine if that performance is or is not meritous, valuable, or significant


  • Login