slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Measuring the Impact of Entertainment-Education Programs PowerPoint Presentation
Download Presentation
Measuring the Impact of Entertainment-Education Programs

Loading in 2 Seconds...

play fullscreen
1 / 39

Measuring the Impact of Entertainment-Education Programs - PowerPoint PPT Presentation


  • 106 Views
  • Uploaded on

D. Lawrence Kincaid, Ph.D. Center for Communication Programs Bloomberg School of Public Health Johns Hopkins University. Measuring the Impact of Entertainment-Education Programs. 4 th International Entertainment-Education Conference September 26-30, 2004 at the Lord Charles Hotel

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Measuring the Impact of Entertainment-Education Programs' - drake


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1
D. Lawrence Kincaid, Ph.D.

Center for Communication Programs

Bloomberg School of Public Health

Johns Hopkins University

Measuring the Impact of Entertainment-Education Programs

4th International Entertainment-Education Conference

September 26-30, 2004 at the Lord Charles Hotel

in Somerset West, South Africa.

overview
Between a “rock and a hard place” . . . without a paddle

Association or cause and effect?

From experimental control by means of research designs

To 8 conditions for causal attribution

Overview
slide3

The biggest limitation to the evaluation of communication

  • Members of the population who watch the program are always different from those who do not watch. (Self-selection bias)
  • These differences are usually related to

the expected outcomes. (behavior, etc.)

  • So, a simple comparison of outcomes by

exposure is biased by the differences

between those exposed and not exposed.

the counter factual dilemma
The Counter-Factual Dilemma
  • How much does a communication program change the behavior of audience members who are exposed to the program compared to what they would have experienced if they had not been exposed?
  • Problem: A perfect solution would require a parallel universe. Used only as an ideal.
  • Randomized experimental design cannot be used for full-coverage programs, so the problem has to be solved with measurement and theory.
slide5

Solutions for Full-Coverage Programs

1. Statistical Controls: Use multiple regression to control for all confounding variables that might affect exposure and the outcome.

2. Simple Matching: Limited to 1-3 variables.

3. Propensity Score Matching: Unlimited number of variables that might affect exposure and the expected outcomes.

4. Theory-based evaluation

slide6

A Practical Alternative

  • The goal is to estimate how much change can be attributed to communication.
  • Establish the criteria for measuring

how much impact can be causally

attributed to your program.

  • Obtain empirical evidence to support

each criterion.

  • Draw an appropriate conclusion.
slide7

First, how do you know who was really exposed? (implementation)

The South African TV Drama, Tsha Tsha

  • No. of characters audience can correctly name from photographs (k=4)
  • No. of correct items of knowledge about the drama (k=8)
  • No. of specific episodes they watched that they can identify(k=26)
slide9

Knowledge of the Drama Independent of the Expected Outcomes (unaided)

  • Who is studying for a UNISA degree?
  • Who steals money from Viwe’s father?
  • Who is Joy’s father?
  • What gift does Viwe give to Andile?
  • Who gets a sexually transmitted infection?
  • What item does Unathi sell to get money for food?
  • Who gives Mimi R1,000 for her business?
  • Who does Cedric send to Johannesburg to deliver dagga?
slide10

Number of the 26 episodes watched

Mark which episodes you have seen:

1. Prince opens a hair salon and interviews girls in the town

2. Mimi and Andile have sex, and she is scared he will wake her grandmother

.

.

13. Funeral of Andile’s mother

.

.

26. A party is organized for Andile to dance in Joburg

slide13

1. Did the outcome change over time?

N = 754; not statistically significant

2% of 10,000,000 = 200,000 youth

slide14

5. Was the observed change large and abrupt?

* Statistically significant: p<0.01

Note: A one-year time interval

slide15

2a. Was the change associated with exposure to communication?

* Statistically significant: p<0.01

slide17

3a. Did exposure occur before the observed change?

Wave 3

No Yes Total

No 405 140 545

Yes 103 105 208

Total 508 245 753

Wave 1

Decided to Remain Faithful to My Partner

Net Increase = 37

Chi-square = 5.63; p<0.05

slide18

3b. Did exposure occur before the observed change?

Decided to Remain Faithful to My Partner

N=753; Chi-square = 46.8, p<.001

slide19

4a. Multiple Regression to Estimate the Independent Effect of Exposure after Controlling for Other Influences *South Africa, 2004

Learned about AIDS on TV

.13

Female

AIDS Attitude

.11

.26

Lagged AIDS attitude (wave 1)

.13

.09

.33

Education

.12

Recall of the Drama

.09

Frequency of TV viewing

.31

-.01

Kwazulu province

residence

* Including lagged attitude means

that the impact of other variables

is on change in attitude.

slide20

4b. Propensity Score Matching to Estimate Effects of Exposure of a Drama on AIDS Attitudes Wave 3 South Africa, 2004

Positive AIDS Attitude

Variables Used to Construct the

Propensity Score for Matching:

1. Age

2. Education

3. Gender

4. Income Level

6. Lagged Aids Attitude

7. Learned about AIDS on TV

8. Province (KwaZulu Natal)

slide21

6. Is there evidence of a dose response?

AIDS Attitude by Level of Recall

7 is the causal inference justified theoretically
7. Is the causal inference justified theoretically?

What is a theory?

  • How communication works.
  • A tool for thinking and action.
theory based evaluation
Theory-Based Evaluation
  • Identify causal pathways

(mediating variables for effect)

  • Demonstrate:
    • Impact of communication on mediating variables
    • Impact of mediating variables on behavioral outcome
slide24

INTENTION

BEHAVIOR

IDEATION MODEL OF COMMUNICATION AND BEHAVIOR

C

O

M

M

UNICATION

SKILLS & KNOWLEDGE

INSTRUCTION

reinforcement

IDEATION

COGNITIVE

Beliefs

Values

Perceived Risk

Subjective Norms

Self-Image

EMOTIONAL

Emotional Response

Empathy

Self-Efficacy

SOCIAL

Support & Influence

Personal Advocacy

DIRECTIVE

Dissemination

Promotion

Prescription

Attitudes

confirmation

NONDIRECTIVE

Dialogue

Counseling

Entertainment

Social Networks

enabling

PUBLIC

Advocacy

Regulation

ENVIRONMENTAL

SUPPORTS & CONSTRAINTS

Source: Adapted from

Kincaid (2000)

slide25

A predictive model of communication & change

Implies simultaneous effect of all influences.

Knowledge

Personal Advocacy

Attitudes

Social Support & Influence

Self-Image

BEHAVIOR

Perceived Risk

Emotions

Self-Efficacy

Implies communication can effect all influences.

Norms

slide27
A Path Model of the Effects of Recall of the Drama on Identification with Boniswa and AIDS Attitudes

Identification with a character in the drama

indirect effects

Attitude towards HIV/AIDS

direct effects

Level of exposure to the drama

direct effects

slide28
7. Path Model of the Effects of Recall of the Drama on Identification with Boniswa and AIDS Attitudes at Wave 3South Africa, 2004

Watches “Days of Our Lives”

.06

Identification with Boniswa

Abstained for a month or more

.09

.34

Learned about AIDS on TV

.13

.12

.13

AIDS Attitude

.11

Female

.26

Lagged AIDS attitude

.13

.33

.09

Education

.12

Recall of the Drama

.09

Frequency of TV viewing

.31

-.01

Kwazulu province

slide29

7. Is the causal inference

justified theoretically?

YES

slide30

8. Are the results consistent with

previous program research?

YES

slide31

Eight Criteria for Causal Attribution

1. Change over time in the expected outcome is observed.

2. An association between that change and program exposure is observed. (correlation)

3. Exposure occurs before the observed change is measured. (time-order)

eight criteria for causal attribution
Eight Criteria for Causal Attribution

4. No evidence of confounding variables that may have accounted for the change.

5. The observed change is abrupt and large.(immediacy and magnitude)

6. The impact increases in proportion to level

or duration of exposure.(Dose response)

slide33

Eight Criteria for Causal Attribution

7.A causal connection is justified.

(causal pathways and theoretical coherence)

8. Consistency with previous program research.(replication with variation)

unfinished work
Unfinished Work

We can still not adequately measure many of the important effects of EE, especially drama.

1. Expanding the perspective and

ways that people make decisions

2. Emotional Involvement (caring, fear)

3. Measuring effects of narrative

measuring the effects of a story
Measuring the effects of a story

The Eagle and the Cocks

  • Two cocks in the farm yard fought to decide who
  • should be master.
  • The loser withdrew to a dark corner, while the victor
  • flew to the roof top and crowed lustily.
  • An eagle spied him from high up in the sky,
  • swooped down, and carried him off.
  • The other cock came out and ruled without a rival.
  • PRIDE COMES BEFORE THE FALL.

Aesop’s Fable (620 B.C.)

slide36

FIN

Thank you!

why programs fail or appear to fail
Why Programs Fail or Appear to Fail
  • Inappropriate theory
  • Appropriate theory, but inadequate
  • application of the theory to the program
  • Appropriate theory, adequately designed,
  • but poorly implemented
  • Appropriate theory, adequately designed,
  • well implemented, but ineffective research
  • Appropriate theory, design, implementation,
  • and research design,but inadequate
  • measurement of the degree of impact
sources of program success
Sources of Program Success
  • Appropriate theory
  • Correct application of the theory
  • to the design of the program
  • Successful implementation
  • Adequate evaluation research
  • Adequate measurement of impact
implications
Implications
  • Five sources are interrelated.
  • If not effective, it’s difficult to know why.
  • Program may be effective, but the
  • research cannot show it.
  • You can only improve if you know what
  • happened and why.
  • There are realistic limits to what we can
  • learn from program evaluation.