eval 6000 foundations of evaluation n.
Skip this Video
Loading SlideShow in 5 Seconds..
EVAL 6000: Foundations of Evaluation PowerPoint Presentation
Download Presentation
EVAL 6000: Foundations of Evaluation

Loading in 2 Seconds...

play fullscreen
1 / 41

EVAL 6000: Foundations of Evaluation - PowerPoint PPT Presentation

  • Uploaded on

EVAL 6000: Foundations of Evaluation. Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2011. Agenda. Valuing theories and theorists Semester review. Evaluation Theory Tree. Criticism and Connoisseurship. Grew out of methods used in art and literary criticism

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'EVAL 6000: Foundations of Evaluation' - elma

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
eval 6000 foundations of evaluation

EVAL 6000: Foundations of Evaluation

Dr. Chris L. S. Coryn

Kristin A. Hobson

Fall 2011

  • Valuing theories and theorists
  • Semester review
criticism and connoisseurship
Criticism and Connoisseurship
  • Grew out of methods used in art and literary criticism
  • Assumes that certain experts are capable of in-depth analysis and evaluation that could not be done in other ways
  • Based on evaluator’s special expertise and sensitivities
  • Methodologically, uses perceptual sensitivities, past experiences, refined insights, and ability to communicate assessment
sensory evaluation
Sensory evaluation
  • Flavor is comprised of
    • Retro nasal aroma
    • Basic taste
    • Chemical feeling factors
  • Aroma
  • Ortho nasal
  • Plug your nose
  • Eat a skittle with your nose plugged
  • Eat a second skittle
  • While chewing, unplug your nose
consumer oriented studies
Consumer-Oriented Studies
  • The evaluator is the enlightened surrogate consumer
  • He or she must draw direct evaluative conclusions about the program being evaluated
  • Evaluation is viewed as the process of determining something’s merit, worth, and significance, with evaluations being the products of that process
consumer oriented studies1
Consumer-Oriented Studies
  • Regards consumer welfare as a program’s primary justification and accords that welfare the same primacy in program evaluation
  • Grounded in a deeply reasoned view of ethics and the common good, together with skills in obtaining and synthesizing pertinent, valid, and reliable information, the evaluator should help developers produce and deliver products and services that are of excellent quality and of great use to consumers
consumer oriented studies2
Consumer-Oriented Studies
  • Advance organizers include societal values, consumers’ needs, costs, and criteria of goodness in the particular evaluation domain
  • Purpose is to judge the relative merits, worth, or significance of the products and services of alternative programs and thereby to help taxpayers, practitioners, and potential recipients make wise choices
consumer oriented studies3
Consumer-Oriented Studies
  • Objectivist in assuming an underlying reality and positing that it is possible, although often extremely difficult, to find best answers
  • Often employ’s ‘goal-free evaluation’
consumer oriented studies4
Consumer-Oriented Studies
  • Employs a wide range of assessment methods
  • Largely, this approach applies Scriven’s general logic of evaluation in arriving at a summative statement
    • Establish criteria of merit
    • Construct standards
    • Measure performance and compare with standards
    • Synthesize and integrate into a judgment of merit or worth
social agenda and advocacy approaches
Social Agenda and Advocacy Approaches
  • Aimed at increasing social justice through program evaluation
  • Seek to ensure that all segments of society have equal access to educational and social opportunities and services
  • Advocate affirmative action to give the disadvantaged preferential treatment through program evaluation
  • If, as many persons have stated, information is power, then these approaches employ program evaluation to empower the disenfranchised
social agenda and advocacy approaches1
Social Agenda and Advocacy Approaches
  • Favor a constructivist orientation and the use of qualitative methods (with the exception of transformative evaluation)
  • Eschew the possibility of finding right or best answers and reflect the philosophy of postmodernism, with its attendant stress on cultural pluralism, moral relativity, and multiple realities
  • Provide for democratic engagement of stakeholders in obtaining and interpreting findings
responsive and client centered studies
Responsive and Client-Centered Studies
  • Client-centered
  • Evaluator works with and for the support of a diverse client group
  • Continuous interaction, and response to, the evaluative needs of clients and stakeholders
  • Findings used primarily for improvement
  • Predominately applies case study methods
constructivist evaluation
Constructivist Evaluation
  • Grounded in a rejection of the experimental philosophy—deeply paradigm driven
  • Sometimes known as 4th generation evaluation—because it goes beyond (1) objectives-based, (2) description, and (3) judgment
  • Intensive participation of stakeholders in the design, conduct, reporting, and application
  • Emphasis on the “constructions” (i.e., social realities) that different stakeholders bring to bear in assessing an evaluand
constructivist evaluation1
Constructivist Evaluation
  • Constructivist evaluation places the evaluators and program stakeholders at the center of the inquiry process, employing all of them as the evaluation’s “human instruments”
  • Constructivist evaluation is seen as a solution to problems in evaluations based on classical experimental design
    • These problems include nonuse of findings, objectification of human beings, the lack of meaningful involvement of stakeholders in evaluations, and the nonuse of evaluative processes by which people make sense of their worlds and those of others
deliberative democratic evaluation
Deliberative Democratic Evaluation
  • Functions within an explicit democratic framework
  • Charges evaluators to uphold democratic principles in reaching defensible conclusions
  • Three main features
    • Democratic participation
    • Dialogue to examine and authenticate stakeholder’s inputs
    • Deliberation to arrive at a defensible assessment of an evaluand’s merit and worth
deliberative democratic evaluation1
Deliberative Democratic Evaluation
  • The approach has many advantages
    • It strives for democratic participation of stakeholders at all stages of the evaluation
    • It seeks to incorporate the views of all interested parties, including insiders and outsiders, disenfranchised persons and groups, as well as those who control the purse strings
    • Meaningful democratic involvement should direct the evaluation to the issues that people care about and make them inclined to respect and use the evaluation findings
    • The approach employs dialogue to examine and authenticate stakeholders’ inputs
customer feedback
Customer Feedback
  • The common practice by numerous commercial enterprises of obtaining and publishing on their Websites consumers’ ratings and (sometimes) narrative reviews of products or services they had supposedly purchased and used
  • In the marketing and communications literature, this approach is often referred to as ‘word of mouth’
transformative evaluation
Transformative Evaluation
  • The transformative evaluation approach represents a worldview, and its accompanying philosophical assumptions, that emerged from the writings of scholars from diverse ethnic and racial groups, people with disabilities, and feminists
  • Evaluators work in contested territory that is laden with pluralistic values and associated with real-life implications for the allocation of resources
transformative evaluation1
Transformative Evaluation
  • From this point of view, knowledge claims are not neutral
    • They are influenced by human interest
    • All knowledge reflects power and social relationships within society
  • Transformative evaluation places a central importance on the lives of marginalized social groups (e.g., women, the poor, ethnic and racial minorities, people with disabilities)
common definition
Common Definition
  • Evaluation is the act or process of determining the merit, worth, or significance of something or the product of that process
evaluative predicates
Evaluative Predicates
  • Merit
    • Intrinsic quality; absent of context and costs
  • Worth
    • Synonymous with value; quality under consideration of costs and context
  • Significance
    • Synonymous with importance; merit and worth in a specific context
evaluative claims
Evaluative Claims
  • Evaluators (ought to) make evaluative claims, or conclusions, beyond the typical empirical or research-based claims of “what’s so?” to inferences or evaluative claims of “so what?”
  • Evaluative claims of the “so what?” variety are couched in value-imbued language such as good or bad, priceless or worthless, and trivial or important, which are subsumed in the vocabularies of merit, worth, and significance
competing definitions
Competing Definitions
  • Evaluation is “the use of social science research procedures to systematically investigate the effectiveness of social intervention programs” (Rossi, Freeman, & Lipsey)
  • Proponents of theory-driven evaluation approaches characterize evaluation as explaining “how and why programs work, for whom, and under what conditions” (Chen)

“Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object”

— William M. K. Trochim

primary purposes
Primary Purposes
  • Formative
    • Evaluation conducted with the intent to improve
  • Summative
    • Evaluation conducted with the intent to inform decision making and/or determine disposition
secondary purposes
Secondary Purposes
  • Developmental
    • To help develop an intervention or program
  • Accountability
    • To hold accountable; usually under summative
  • Monitoring
    • To assess implementation and gauge progress toward a desired end
  • Ascriptive
    • Merely for the sake of knowing
basic functional forms
Basic Functional Forms
  • Process evaluation
    • Assessment of everything that occurs prior to true outcomes
  • Outcome evaluation
    • Assessment of an evaluand’s effects
  • Cost evaluation
    • Assessment of monetary and non-monetary costs, direct and indirect costs, and actual and opportunity costs
potential direct users
Potential Direct Users
  • People who have authority over a program
  • People who have a direct responsibility for a program
  • People who are the intended beneficiaries of a program or who are impacted by a program (direct and indirect)
  • People disadvantaged by a program
  • Instrumental
    • Immediate decision making or action taking
  • Conceptual
    • Impact on the way people think about an evaluand
  • Process
    • Changes that occur as the result of an evaluation process
  • Persuasive
    • Decision makers use an evaluation to move others to support a decision
uses and misuses
Uses and Misuses



Ideal Use

Mistaken Use

(incompetence, uncritical acceptance, unawareness)

Mischievous Use

(manipulation, coercion)

Instrumental Use

Conceptual Use

Persuasive Use

Legitimate Use


Rational Non-Use

Political Non-Use


(inappropriate suppression of findings)

Unjustified Non-Use

Justified Non-Use


politics of evaluation
Politics of Evaluation
  • Evaluation is unavoidably implicated in matters of politics
    • Programs are created and maintained by political forces
    • Higher echelons of government, which make decisions about programs, are embedded in politics
    • The very act of evaluation has political connotations
auditing versus evaluation
Auditing versus Evaluation
  • Auditing is a procedure in which a third party systematically examines the evidence of adherence to some practice to a set of norms or standards for that practice and issues a professional opinion
  • This general idea spawned the notion of metaevaluation (Scriven)
research versus evaluation
Research versus Evaluation
  • The application of scientific logic and methods to social phenomena undertaken to advance knowledge or theoretical understanding (basic research) or with the intent of applying the results to some specific problem (applied research)
evaluation theory
Evaluation Theory
  • Evaluation theories describe and prescribe what evaluators do or should do when conducting evaluations and are mostly normative in origin
  • They specify such things as evaluation purposes, users and uses, who participates in the evaluation process and to what extent, general activities or strategies, methods choices, and roles and responsibilities of the evaluator, among others
logic of evaluation
Logic of Evaluation
  • Scriven’s logic of evaluation is the closest to a meta-theory of evaluation
    • Establishing criteria
      • On what dimensions must the evaluand do well?
    • Constructing standards
      • How well should the evaluand perform?
    • Measuring performance and comparing with standards
      • How well did the evaluand perform?
    • Synthesizing and integrating information/data into a judgment of merit or worth
      • What is the merit or worth of the evaluand?
general premises
General Premises
  • Factual premises
    • The nature, performance, or impact of an evaluand or evaluee
    • Roughly equivalent to description (“what’s so?”)
  • Value premises
    • The properties or characteristics (i.e., criteria and standards) which typify a good, valuable, or important evaluand or evaluee of a particular class or type in a particular context
value premises
Value Premises
  • General values
    • The merit-defining criteria by which an evaluand or evaluee is evaluated; the properties or characteristics which define a ‘good’ or ‘valuable’ evaluand or evaluee
  • Specific values
    • The standards (i.e., levels of performance; usually an ordered set of categories) which are applied and by which performance is upheld, in order to determine if that performance is or is not meritous, valuable, or significant