theory experimental design l.
Skip this Video
Loading SlideShow in 5 Seconds..
Theory & Experimental Design PowerPoint Presentation
Download Presentation
Theory & Experimental Design

Loading in 2 Seconds...

play fullscreen
1 / 103

Theory & Experimental Design - PowerPoint PPT Presentation

  • Uploaded on

Theory & Experimental Design. Part II. FTT & Political Science. Two generations (although overlap time wise): First – testing equilibrium & non-equilibrium predictions of social choice theory Second – testing more applied models, typically with greater institutional (political) detail.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Theory & Experimental Design' - Leo

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
ftt political science
FTT & Political Science
  • Two generations (although overlap time wise):
    • First – testing equilibrium & non-equilibrium predictions of social choice theory
    • Second – testing more applied models, typically with greater institutional (political) detail.
  • Mirror evolution of formal models in discipline
first generation tests on elections committees tests of spatial voting models
First Generation Tests on Elections & Committees – Tests of Spatial Voting Models
  • This work is reviewed in
    • McKelvey, Richard D. & Peter C. Ordeshook, 1990, “A Decade of Experimental Research on Spatial Models of Elections and Comittees,” in James M. Enelow & Melvin Hinich, eds., Advances in the Spatial Theory of Voting, Cambridge: Cambridge U. Press.
first generation tests on committee voting
First Generation Tests on Committee Voting
  • Goal of initial work on committees –
    • see extent disequilibrium really happened in lab.
  • Not so easy to avoid equilibrium creating conditions not suggested by theory.
  • For example, if subjects know (or suspect) experiment has an “end” or consider time spent as costly, may be motivated to agree on a choice.
first generation tests on generic voting games
First Generation Testson Generic Voting Games
  • While fundamental – largely become province of social choice theorists & theoretical side dominates
  • Main emphasis what is necessary get equilibrium, considering axioms of social choice theory & implications.
  • Social Choice & Welfare (both society & journal) – main avenues, mainly normative
example of modern first generation like voting experiment
Example of “Modern” “First Generation Like” Voting Experiment
  • “Voting Games and Computational Complexity,” Glenn W. Harrison & Tanga McDaniel, working paper
voting rule experiment
Voting Rule Experiment
  • Authors confront Gibbard-Satterthwaite Theorem – loosely stated – only voting rule that is “strategy proof” for all possible preference profiles is dictatorial.
  • “strategy proof” – when choices people make under voting rule are truthful – reflect true preferences.
  • Consider an example – 3 choices, x, y, & z, & 3 types of voters, A (40 voters), B (20 voters), & C (40 voters)
example of strategic voting
Example of Strategic Voting

Who will win this election? Not obvious.

If voters A, B, & C all voted for their most preferred choice

(truth telling), & we use majority rule, then a tie between x & z.

example of strategic voting9
Example of Strategic Voting

But is that what we would want as a society? Normatively?

Condorcet argued that most preferred choice is one who would

Defeat or tie all others in pairwise (binary) contests.

example of strategic voting10
Example of Strategic Voting

Who is the Condorcet winner?

Binary choice between x & y, A votes for x, B & C for y, y wins.

Binary choice between y & z, A & B for y, C for z, y wins.

y is the Condorcet winner.

example of strategic voting11
Example of Strategic Voting

Majority rule is definitely not “strategy proof” & in fact,

normatively we might prefer that voters “strategize”

& positively expect them to – Making Votes Count

goal of voting experiment
Goal of Voting Experiment
  • Harrison & McDaniel contend that some voting rules may be more difficult (in a computational sense) to manipulate than others.
  • They want to test their hypothesis in the lab.
voting rule experiment goal
Voting Rule Experiment: Goal
  • H & M test a voting rule in laboratory they believe is
    • Easy to explain to subjects
    • Easy to implement
    • Difficult for subjects to strategically manipulate (because of unspecified cognitive limits)
voting rule experiment goal14
Voting Rule Experiment: Goal
  • Theory testing or fact finding?
    • While voting rule is “script” as in other formal theory experimental tests, H & M expect something not modelling (cognitive limits of subjects) is important & are “fact finding.”
  • Also, some “policy pre-testing”
  • Nevertheless – design of experiment is “scripted” by theory
proposed voting rule
Proposed Voting Rule
  • One voting rule that might be good is to always select Condorcet winner – in fact this has been proven to be strategy proof.
condorcet winner voting rule
Condorcet Winner Voting Rule

Can voters manipulate this voting rule getting a

more preferred outcome

by misrepresenting their preferences?

condorcet winner voting rule17
Condorcet Winner Voting Rule

Who is the Condorcet winner now?

proposed voting rule18
Proposed Voting Rule
  • problem –Condorcet winner not always exsit.
  • Extension of Condorcet rule by Young – find non-cyclical ranking with most support of voters.
  • Turns out solution found by solving linear programming problem
  • Condorcet Consistent Voting Rule – basically chooses Condorcet winner if one exists, if not, non-cyclical ranking with greatest support.
condorcet consistent voting rule
Condorcet Consistent Voting Rule

Who would win? In this example C voters’ ranking would

be maximal, note similarity w/ dictatorship in this example

-- but rule not same as dictatorship.

computation voting rule
Computation & Voting Rule
  • strategic voting under Condorcet Consistent voting rule requires complex computations when # of alternatives is large –
    • for n alternatives there are n! non-cyclic rankings.
  • Thus while not strategy proof in theory, contend behaviorally incentive compatiblewith truth telling (non strategic voting).
  • Point of Experiment – is this true?
inducing preferences in voting experiments
Inducing Preferences in Voting Experiments
  • Usually money
  • Tell subjects will pay based on which alternative, x,y,z is chosen & have subjects vote.
  • Can measure extent of strategic voting by comparing choices to induced preferences.
home grown preferences
“Home Grown” Preferences
  • Harrison & McDaniel use “home grown” preferences
  • Subjects given list of CD’s, grouped in categories
  • Subjects vote over which category for group, then each picks a CD from the list of 10 in category
  • Difficulty – how measure strategic voting?
how measure strategic voting
How Measure Strategic Voting?
  • Use “control” treatment – subjects choose CD’s under a “random dictator” voting rule
  • But to be sure instruct subjects in this treatment only on advantages of telling truth – non neutral instructions
  • Note importance of random assignment.
  • Are home grown preferences desirable here?
heterogeneity of preferences
Heterogeneity of Preferences
  • strategic voting “harder” when preferences of voters more heterogeneous.
  • Vary heterogeneity :
    • Simple treatment – music categories of Jazz/Easy Listening, Classical, R&B, Rock, C&W.
    • Complex treatment – music categories of Jazz/Easy Listening, Classical, Heavy Metal, Rap, C&W.
  • Contend most subjects prefer R&B or Rock.
table 1 musical categories cd s category a jazz easy listening
Table 1: Musical Categories & CD’sCategory A: Jazz/Easy Listening
  • (1) Najee, “Share My World”
  • (2) Kenny G, “Breathless”
  • (3) Art Porter, “Undercover”
  • (4) Russ Freeman & the Rippingtons, “Sahara”
  • (5) Tony Bennet, “Unplugged”
  • (6) George Howard, “A Home Far Away”
  • (7) Enigma 2, “The Cross of Change”
  • (8) Billy Joe Walker, “Life is Good”
  • (9) Barry Manilow , “Singing in the Big Bands”
  • (10) Nat King Cole, “The Greatest Hits”
category b classical
Category B: Classical
  • (1) John Williams & the Boston Pops Orchestra, “It Don’t Mean a Thing if it ain’t got that Swing”
  • (2) Vivaldi, “The 4 Seasons” Gil Shattam Orpheus: Fritz Kreisler.
  • (3) Mahler, Symphony #5. The New York Philharmonic: Leonard Bernstein.
  • (4) Yo Yo Ma, The New York Album. Baltimore Symphony Orchestra: David Zinman.
  • (5) Handel, “Messiah” Atlanta Symphony Orchestra & Chamber Chorus: Robert Shaw.
  • (6) Cecilia Bartoli, “Mozart Portraits” Vienna Chamber Orchestra: György Fischer.
  • (7) Van Clyburn in Moscow. Brahms Rachmaninoff. M oscow Philharmonic Orchestra: Kiril Konorashin.
  • (8) Kiri, “Her Greatest Hits Live” London Symphony Orchestra: Steven Barlow.
  • (9) Tchaikovsky, “Nutcracker” London Symphony Orchestra: S ir Charles Mackerras.
  • (10) The Best of Wolfgang Amadeus Mozart. Academy of St. Martin-in-the-fields: Neville Marriner.
category c heavy metal
Category C: Heavy Metal
  • (1) Pantera, “Far Beyond Driven”
  • (2) Queensryche, “Promised Land”
  • (3) Magadeth, “Youthanasia”
  • (4) Motley Crüe, “Motley Crue” (featuring Hooligan’s Holiday)
  • (5) Mother Tongue “Mother Tongue”
  • (6) Obituary, “World Demise”
  • (7) Jackyl, “Push Comes to Shove”
  • (8) Alice in Chains, “Jar o f Flies”
  • (9) Alice Cooper, “The Last Temptation”
  • (10) Cinderella, “Still Climbing”
category d rap
Category D: Rap
  • (1) Pete Rock and CL Smooth, “The Main Ingredient”
  • (2) Lighter Shade of Brown, “Layin in the Cut”
  • (3) Craig Mack, “Project: Funk Da World”
  • (4) Ghetto Mafia, “Draw the Line”
  • (5) Common Sense, “Resurrection”
  • (6) Stevie B, “Funky Melody”
  • (7) Salt-n-Pepa, “Very Necessary”
  • (8) j. Little, “Puttin’ it Down”
  • (9) Celly Gel, “Heat 4 Yo Azz”
  • (10) Hammer, “The Funky Headhunter”
category e country western
Category E: Country & Western
  • (1) Garth Brooks, “In Pieces”
  • (2) George Ducas, “George Ducas”
  • (3) Shenandoah, “In the Vicinity of the Hearth”
  • (4) Chris Ledoux, “Haywire”
  • (5) Willie Nelson, “Healing Hands of Time”
  • (6) Vince Gill, “When Love Finds You”
  • (7) Pam Tillis, “Sweetheart’s Dance”
  • (8) Noah Gordon, “I Need a Break”
  • (9) Rodney Crowell, “Let the Picture Paint Itself”
  • (10) Ricky Lynn Gregg, “Get a Little Closer”
information to subjects
Information to Subjects
  • Varied information gave subjects:
    • Information Treatment – specifics of CC voting rule spelled out to subjects & examples supplied
    • No Information Treatment – subjects only told that the social ranking chosen would be the one which would most likely receive the support of a majority of the voters.
  • Why vary this?
  • Policy Pre-testing . . .
results of voting rule experiment
Results of Voting Rule Experiment
  • Gist – found under simple preference profile significant difference between rankings in CC & RD.
  • No significant difference between rankings in CC & RD under tough preference profile.
  • Moreover, information matters only in simple preference profile experiments.
  • Results support argument that when preferences are “tough” to figure out, CC does elicit truth telling.
first generation research summary
First Generation Research: Summary
  • Voting Rule Experiment example of an experimental test of research from social choice theory –
    • like first generation of FTT in political science.
  • While experiment interesting & results important, most political scientists not think FTT as in political science today.
  • Why?
impact of disequilibrium results on formal theory in political science
Impact of Disequilibrium Results on Formal Theory in Political Science
  • Economists can frequently start from wellaccepted equilibrium models, & then do comparative statics bystandard techniques.
  • Political theory not have well acceptedequilibrium models to start from.
  • Theory must incorporatedetails of situation.
impact of disequilibrium results on formal theory in political science35
Impact of Disequilibrium Results on Formal Theory in Political Science
  • explicitly model role of
    • information,
    • repetition
    • & institutions
    • usually accompanied by increasing use of non-cooperative game theory,
    • incompleteinformation,
    • & explicit specification of extensive forms.
  • The “New Institutionalism”
impact of disequilibrium results on formal theory in political science36
Impact of Disequilibrium Results on Formal Theory in Political Science
  • Anothertrend – evolutionary & agent based models.
  • Different view of behavior – individuals programmed to behave in certain ways & only changebehavior through replacement or imitation (e.g. Bendor, Diermeier, & Ting)
  • Complex processes literature, etc.
  • Difficult for laboratory experiments – long term processes of evolution unlikely in single experiment.
experiments on classic games political science
Experiments on Classic Games & Political Science
  • Classic games often building blocks in formal work in political science.
  • Example – bargaining games ultimatum & dictator games add together to get Baron/Ferejohn legislative bargaining game
  • Example – voting turnout like public good/pd combined
turnout as combined public good pd
Turnout as Combined Public Good/PD
  • Think of a two candidate election w/ two groups of supporters (teams or political parties).
  • Each group member individually decides whether or not to vote, paying individualized cost to voting.
  • Group with most voters wins group payoff distributed equally to all group members whether voted or not.
  • Team turnout game combination of public good game (within a team) & a prisoner’s dilemma game (between teams).
turnout experiment theory
Turnout Experiment: Theory
  • Turnout modelled this way by Palfrey & Rosenthal, 1983, “A Strategic Calculus of Voting,” Public Choice & in APSR, 1984.
  • Showed equilibria exist with positive turnout.
  • Model tested experimentally by Schram and Sonnemans, International Journal of Game Theory, 1996.
  • Good example of ways formal theory testing works with more complex game.
turnout experiment design
Turnout Experiment: Design
  • Subjects split into 2 groups of 6 each, labelled yellow & blue.
  • Each subject had to decide whether to buy an imaginary disc.
  • Price of a disc was common knowledge & equal for everyone.
  • # of discs bought by group determined payoffs.
  • Payoffs equal for everyone within a group.
  • Repeated for 20 periods.
turnout experiment design41
Turnout Experiment: Design
  • Two payoff schedules, representing winner take all (WIN) & proportional representation (PR).
  • In WIN, each group member bought most discs received payoff of 2.5 Dutch gilders & other group received zero, ties broken randomly.
  • In PR, payoff was proportional to turnout within group – # of discs bought in one’s group was divided by total number bought & multiplied by 2.22 Dutch gilders.
  • Price of disc – 1 gilder (WIN), 0.75 gilders (PR)
turnout experiment predictions
Turnout Experiment: Predictions
  • Nash equilibria of games in pure strategies (one shot):
    • In PR, one disc bought by each group.
    • In WIN, 6 discs bought by each group.
    • (turnout theoretically higher in WIN, why? – relationship prediction)
  • Quasi-symmetric mixed strategy equilibria:
    • In PR, all subjects buy a disc with probability 0.098.
    • In WIN – two equilibria
      • All subjects buy with probability 0.051
      • All subjects buy with probability 0.949
turnout experiment results
Turnout Experiment: Results
  • Find comparative static predictions are supported (relationship predictions)
  • But point predictions not.
  • Schram & Sonnemans in Journal of Economic Psychology, 1996 compare analysis with hypotheses derived from a model of turnout that incorporates group pressure as explanation of turnout.
  • Find some support for group model, although experiment not an explicit test of model (since designed to test Palfrey/Rosenthal)
ftt experimental design
FTT & Experimental Design
  • Game directly from mechanics of Palfrey/Rosenthal model (script).
  • not “described” to subjects as voting situation with candidates as in election (frame).
  • Why?
  • Advantage –
    • experiment tests “theory” of participation without “baggage” subjects may bring about voting as an act –
    • can serve as “baseline” results to experiments where act is described as voting w/ candidates etc.
  • Disadvantage – decreases “external validity”?
  • Is this deception?
ftt experimental design45
FTT & Experimental Design
  • Palfrey/Rosenthal –
    • general model without specific values – for experiment specific values must be set for payoffs & cost of participation.
  • Experimenters must solve model for equilibrium predictions with specific values.
  • Palfrey/Rosenthal model had no PR, only WIN
  • experimenters take model, solved it for particular values, plus made modifications to model.
ftt experimental design46
FTT & Experimental Design
  • Unless working with simple formal theory, usually modifications/limitations needed in testing a formal theory
  • Difficult to truly fit theory as originally devised even in laboratory with a lot of control
  • Still often easier than w/o naturally occurring data . . .
ftt experimental design47
FTT & Experimental Design
  • Probably more difficult to fit theory as script for experiment for political science models than for economists.
  • Why? more institutional detail, more applied, & thus more bells & whistles to either try to put in or simplify.
fitting design to theory example of difficulties
Fitting Design to Theory: Example of Difficulties
  • choice variable often a continuous – makes solving models easier & sometimes necessary for solutions.
  • two candidate competition over a unidimensional policy space, assume candidates can choose any point –an infinite choice set.
  • But suppose wanted to test this – if tell subject choose number between 1 & 10 & tell subject any fraction is acceptable, is subject truly “thinking” continuously?
theory testing review
Theory Testing: Review
  • Psychological or Social Psychological Theories
    • Typically non formal, i.e.
      • assumptions underlying theory stated verbally
      • hypotheses about variables posited.
    • Equilibrium predictions rarely derived.
    • Usually decision-theoretic
    • Focused often on process of choice & internal process of mind.
political psychologists social psychologists formal theory
Political Psychologists, Social Psychologists & Formal Theory
  • Some political psychology or social psychology research involves formal theories.
  • Prospect theory (Kahneman & Tversky)
  • Has been subject of political science experiments:
    • Quattrone, George A. & Amos Tversky, “Contrasting Rational and Psychological Analyses of Political Choice,” American Political Science Review 82, 3, Sept. 1988, pages 720-36.
political psychologists social psychologists formal theory52
Political Psychologists, Social Psychologists & Formal Theory
  • Some involve testing hypotheses loosely derived from assumptions behind rational choice based formal theories.
  • Often little of both, testing a non formal theory based on research from psychology & social psychology in contrast to hypothesis about rational choice implication for situation.
psychological social psychological theory testing experiments ptt in political science
Psychological & Social Psychological Theory Testing Experiments (PTT) in Political Science
  • PTT in political science builds on experimental research in psychology & social psychology generally.
  • Often application to political science context theories arising from psychology & social psychology.
ptt in political science
PTT in Political Science
  • traditional approach of hypothesis testing through random assignment, control (baselines), & manipulation.
  • more of a script (I.e. is placed in a political context for external validity reasons) than in psych, however still usually less of a script than FTT.
example of ppt
Example of PPT
  • Taber, Charles & Milton Lodge, “Motivated Skepticism in the Evaluation of Political Beliefs,” 2000
  • Results from experiments in psychology – individuals “are prone to accept at face value evidence that is congruent with prior beliefs & denigrate or hyper-critically evaluate evidence that is contrary to their priors.”
  • Taber & Lodge evaluate in context of theory of political reasoning.
theory of motivated reasoning
Theory of Motivated Reasoning
  • Assume all reasoning is motivated.
  • Two types of motivation
    • Accuracy Goals – seek information to be correct
    • Partisan Goals – reason to defend a prior (psychology literature calls these directional goals)
  • Continuum, pure “rationalist” to pure partisan
three mechanisms assumed underlie motivated reasoning
Three Mechanisms Assumed Underlie Motivated Reasoning
  • Hot cognition hypothesis – social concepts evaluated in past “affectively” charged (negatively or positively) – confronted with concept recall “affect” – feeling.
  • On-line processing– every time exposed to concept, one’s summary affective evaluation recalled & updated.
  • How do I feel? Heuristic – OL tally stored with concept is automatically available for future information processing.
experimental hypotheses
Experimental Hypotheses
  • Hypothesis 1: a prior attitude effect– people who feel strongly about an issue, even when encouraged to be objective, will evaluate supportive arguments as stronger & more compelling than contrary arguments
  • Hypothesis 2: a disconfirmation bias– people will spend more time & cognitive resources denigrating & counter arguing attitudinally incongruent than congruent arguments.
  • Hypothesis 3: a confirmation bias – when free to choose information, people will seek out confirming over disconfirming information.
experimental hypotheses59
Experimental Hypotheses
  • Contend above three hypotheses imply fourth:
  • Hypothesis 4: attitude polarization– attitudes will become more extreme even when confronted with a balanced set of pro & con arguments.
two further experimental hypotheses
Two Further Experimental Hypotheses
  • Hypothesis 5: an attitude strength effect– those with strongest policy attitudes more likely to have biases above.
  • Hypothesis 6: a sophistication effect– political sophisticates, because have greater ammunition to counter argue more likely to have biases above.
experimental design61
Experimental Design
  • Subjects recruited from intro political science classes & given class credit for participation
  • Subjects seated at computers, told participating in study of public opinion.
  • Completed two tasks
  • Note – subjects told to “put feelings aside” – “to be objective” etc.
experimental design first task
Experimental Design: First Task
  • Asked attitude questions on # of issues including gun control or affirmative action, randomly assigned.
  • Given chance to “practice” on information board w/ arguments pro & con on same issue –arguments subjects examined & time spent on each is measured “secretly.” – is this deception?
  • Reevaluated attitudes & asked standard demographic information & measures of political sophistication
  • Designed to test confirmation bias.
experimental design second task
Experimental Design: Second Task
  • Again, asked attitude questions (on gun control if assigned affirmative action, & vice versa)
  • Asked rate strength of 8 arguments, 4 pro & 4 con, presented in random order.
  • Followed by attitude battery & memory recog. task
  • completed thoughts listing task for 2 pro & 2 con arguments
  • Designed to test disconfirmation bias.
experimental design misc issues
Experimental Design: Misc. Issues
  • Arguments gathered from interest groups.
  • Edited so similar in sentence length, sentences per argument, reading level etc.
  • One problem with experimental design – where is control or baseline experiment?
results generally support hypotheses
Results Generally Support Hypotheses
  • Subjects see arguments congruent with beliefs as stronger – prior attitude effect
    • Effect greater for sophisticated & extremists.
  • Subjects took more time to read incongruent arguments – disconfirmation bias
    • Effect greater for sophisticated & extremes
    • When asked thoughts – tended to be against incongruent arguments
  • Some evidence of polarization (although unclear how strong)
another example of ptt
Another Example of PTT
  • Huddy, Leonie & Nayda Terkildsen, “Gender Stereotypes and the Perception of Male and Female Candidates,” American Journal of Political Science, 37, 1 (Feb. 1993), 119-147.
  • Research questions – do voters use gender stereotypes in judging political candidates & if so what type.
  • Two types of gender stereotyping investigated:
    • Gender-trait stereotypes – personality traits seen as gender linked.
    • Belief-trait stereotypes – political outlooks seen as gender linked.
another example of ptt67
Another Example of PTT
  • Trait theory predicts –
    • candidates with masculine traits more competent on military, crime,& defense issues regardless of gender.
    • candidates with feminine traits more competent on compassion issues regardless of gender.
  • Belief theory predicts – inferences about ideology leads to competency ratings
    • Females –Liberals more competent on compassion issues
    • Males – Conservatives better on military, crime, & defense
experimental results
Experimental Results
  • Used students at Stony Brook
  • Find more support for gender traits explaining evaluations of candidates, some minor support for belief stereotypes.
ptt design v ftt design
PTT Design v. FTT Design
  • Less script for subjects to follow – I.e. told somewhat vague & slightly false information about purpose of experiment (some small deception)
    • instructions not part of paper, almost always appendix to FTT experiments.
  • Focus on individual rather than group behavior.
  • No repetition.
  • Choices subjects made (in responding to questions, choosing information to look at & length of time) not related to payment for participation
purpose of scripts in ftt
Purpose of Scripts in FTT
  • Scripts (instructions) supply descriptions of players, action choices, & possible payoffs.
  • FTT experimenters ask participants to enact scripts.
  • In Schram & Sonneman’s experiment, subjects told would be purchasing a disk, reward to them if their group purchased more disks, etc.
advantages of scripts
Advantages of Scripts
  • Scripts increase replicability
  • Allow researcher to trace sometimes subtle influence of institutional details.
  • When experiment is less scripted or not believable,
    • subject guessing what experiment designed to do – “ad-lib” – introducing lower control over variables in experiment.
  • Serious problem if experiment involves complex tasks.
disadvantage of scripts
Disadvantage of Scripts
  • If keeping subjects ignorant of purpose is important, using a script makes deception more explicit.
  • If subjects know time spent reading an argument is measured etc., behavior may be affected
  • By being less precise about experiment’s purpose, some deception is avoided – omitting truth different from lying about truth.
  • Thus in simple decision-making experiments detailed script may not be necessary & harmful.
do scripts make a difference
Do Scripts Make a Difference?
  • Hertwig & Ortmann, 2001, “Experimental Practices in Economics: A Methodological Challenge for Psychologists,” Behavioral and Brain Sciences 24(4),
  • Consider two investigations of hindsight bias reviewed by Hertwig & Ortmann:
    • Camerer, Loewenstein, & Weber (1989) “The curse of knowledge in economic setting: An experimental analysis.” Journal of Political Economy 97:1232-1255.
    • Davies, M. F. (1992) “Field dependence and hindsight bias: Cognitive restructuring and the generation of reasons.” Journal of Research in Personality 26:58-74.
do scripts make a difference75
Do Scripts Make a Difference?
  • In CLW: “ ‘uninformed’ group of participants guessed future earnings of real companies based on information such as previous annual earnings per share.
  • An ‘informed’ group of participants (told actual earnings) then traded assets that paid dividends equal to earnings predicted by uninformed group.
  • Participants in both groups provided with precise script.
  • Those in uninformed group given role (script) of market analyst faced with task of predicting future dividends of various companies.
do scripts make a difference76
Do Scripts Make a Difference?
  • Those in informed group assigned role of trader: knew dividend was determined by uninformed group’s predictions.
  • Thus, to price assets optimally (and avoid hindsight bias), ‘traders’ had to predict prediction of ‘analysts’ accurately, to ignore their knowledge of actual dividends.
  • Eventually, traders traded assets to others in actual double-oral auctions, in which ‘buyers & sellers shouted out bids or offers at which they were willing to buy or sell.
  • When a bid and offer matched, a trade took place’ (p. 1236).
do scripts make a difference77
Do Scripts Make a Difference?
  • “In Davies participants given series of assertions & asked to rate truth of each,
  • then given feedback (i.e., truth values of assertions)
  • & later asked to recall original judgment.
  • In contrast to CLR, Davies did not assign specific roles to participants or provide them with precise script.
do scripts make a difference78
Do Scripts Make a Difference?
  • first stage of study, during which participants rated assertions for their truth,
  • was described to participants as ‘involving evaluation of college students’ knowledge’ (Davies 1992, p. 61),
  • & were told that recollection stage ‘concerned people’s ability to remember or recreate a previous state of knowledge’ (Davies 1992, p. 61).”
do scripts make a difference79
Do Scripts Make a Difference?
  • CLW compared amount of hindsight bias in predictions of participants who enacted role of trader (i.e., who actually traded assets in double-oral auction)
  • to bias in predictions made by another group of participants who did not enact role of trader.
  • Goal of two groups same:
  • predict average prediction of uninformed group given companies’ actual earnings.
do scripts make a difference80
Do Scripts Make a Difference?
  • Both groups received incentives for making correct predictions.
  • CLW reported participants in both conditions exhibited some hindsight bias,
  • but enactment of trader role reduced bias by about half.
another example wason task
Another Example: Wason Task
  • Subjects shown 4 cards displaying symbols such as T, J, 4, & 8
  • & are given a conditional rule about cards,
  • such as “If there is a T on one side of the card [antecedent P], then there is a 4 on the other side of the card [consequent Q].”
  • Participants told each card has letter on one side & number on other.
another example wason task82
Another Example: Wason Task
  • asked which cards would need to turn over to discover whether conditional rule is true or false.
  • Typical result, replicated many times,
  • few participants (10%) give answer prescribed by propositional logic: T and 8 (P & not-Q).
wason selection task
Wason Selection Task
  • Most choose either T (P) alone or T and 4 (P & Q).
  • ‘errors’ seen as reflections of confirmation bias, matching bias, & availability heuristic.
  • However, putting it in a social context, increases percentage of logically correct answers.
wason selection task84
Wason Selection Task
  • police officer checking whether people conform to certain rules:
  • in the context of a drinking age law (‘If someone is drinking beer [P], then they must be over 19 years of age [Q]’),
  • 74% of participants gave the logical P & not-Q response.
wason selection task85
Wason Selection Task
  • Gigerenzer and Hug (1992) way social context affects reasoning depends on perspective participants cued.
  • For instance, rule ‘If an employee works on weekend, then that person gets a day off during week’ depend on whether it is seen from perspective of an employer or of an employee.
  • Employee role dominant answer was P & not-Q (75%);
  • Employer role dominant response was not-P & Q (61)
do scripts matter
Do Scripts Matter?
  • Scripts form of framing & matter in experiments – especially when measuring incidents of biases in human decision making.
  • Subjects in Taber & Lodge experiment encouraged to be unbiased & objective.
  • Would it have mattered if they had been told to play a “role” of a political decision maker?
final word on scripts
Final Word on Scripts
  • In FTT, do we “tell” subjects what to do?
  • Try for neutrality – how experiment works, role to play, but without how to play role.
  • That is, not pointing out their supposedly optimal strategy, unless it is purpose of experiment (as in Harrison & McDaniel’s random dictator treatment).
  • Is there a difference between “framing” by giving context to a role & such prompting a subject directly to use certain strategies?
individual v group behavior
Individual v. Group Behavior
  • One reason FTT uses scripts strategic behaviors.
  • Partly reflects difference in research questions
  • Most PTT researchers extremely interested in process of mind in making choices, psychological mechanics behind choice.
individual v group behavior89
Individual v. Group Behavior
  • Can see in Taber & Lodge experiment, emphasis on mechanics -- & On Line theory, how reasoning works, etc.
  • Emphasis on process desire to find what social psychologists call “mediators” or “generative mechanisms” explain how or why manipulated treatments have an impact on choice
  • Also desire to find “moderators” or factors that affect likelihood of effects demonstrated in experiment – similar to fact finding in FTT.
individual v group behavior90
Individual v. Group Behavior
  • Focus of PTT on individual behavior may affect results – CLW’s hindsight experiments may show less error than traditional non scripted tests because subjects were involved in a game situation.
  • One argument – decision-theoretic experiments appropriate for studying voters & public opinion but not appropriate for understanding behavior of elites.
  • Ideally, both levels of analysis are useful.
role of repetition
Role of Repetition
  • Formal theory testers use repetition partly to increase data – note not all use repetition (trust & bargaining games often exceptions)
  • However, repetition also:
    • Gives chance to adapt to environment, to accrue experience with experimental setting & procedure
    • Affords opportunity to learn how own choices interact with other players.
role of repetition92
Role of Repetition
  • Two types of learning during repetition
    • Learning about laboratory environment & task
    • Learning about possible strategic aspects of decision situation. In fact, in some game theoretic situations potential of repeated play implies different choices.
  • Vary rarely do studies by psychologists & social psychologists have repetition & learning
  • Hertwig & Ortmann estimate only 10% provide any feedback for learning to subjects during experiment.
role of repetition93
Role of Repetition
  • Hertwig & Ortmann show repetition significantly decreases biases & errors
  • found in decision-making experiments, mere “practice” without feedback can improve subjects’ performance.
  • For example, research on preference reversals shows repetition eliminates such reversals.
  • For another example, see Daniel Friedman, “Monty Hall’s Three Doors: Construction and Deconstruction of a Choice Anomaly,” American Economic Review, 88:4, 933-946 (Sept. 1998)
argument against repetition
Argument Against Repetition
  • Individuals in “real world” situations not always get opportunity for repetition – repetition decreases external validity of experiment
    • counter argument – in real world have sources of advice from experienced individuals
  • In many trust game experiments & others, often no repetition.
  • Key is research question – if believe that inexperienced subjects important for question, then repetition can distort results.
deception in ptt ftt
Deception in PTT & FTT
  • While our example did not contain a lot of deception (more lack of information than outright deception) – deception is much more common in PTT than in FTT.
  • Hertwig & Ortmann report deception in experimental articles in top ranked journal in social psychology, Journal of Personality and Social Psychology (JPSP), averages between 31% & 47%,
  • Hardly any FTT experiments have deception (can think of only one recent one).
arguments against deception negative externality
Arguments Against Deception: Negative Externality
  • Private benefits > private costs but social costs > social benefits
  • Hertwig & Ortmann: Subjects’ expectation will not be deceived (by experimenter) is a common good that would be depleted (contaminated) quickly if deception was allowed & decision left to each experimenter.
  • FTT not trust experimenters to make unbiased analysis of (private) benefits of deception & its (public) costs.
  • APA requirement to reveal after makes worse.
arguments against deception ethics
Arguments Against Deception: Ethics
  • American Psychological Association (APA) ethics guidelines (APA 1992, p. 1609) propose to employ deception as a last-resort strategy, to be used only after careful weighing of benefits &costs
  • But frequent use of deception in some areas of psychology seems to confirm economists’ fear.
  • Second argument against deception – ethical argument – which arises from long history of questionable experiments (Tuskegee, etc.)
arguments for deception only way to measure subject behavior
Arguments For Deception: Only Way to Measure Subject Behavior
  • H&O: If subjects aware of true purpose, might respond strategically & investigator might lose experimental control.
  • For instance, one might expect participants to “bend over backwards” (Kimmel 1996, p. 68) to show how accepting they are of members of other races if they know that they are participating in a study of racial prejudices.
arguments for deception used to create situations not natural
Arguments For Deception: Used to Create Situations Not “Natural”
  • Deception can be used to produce situations of special interest that are unlikely to arise naturally (e.g., an emergency situation in which bystander effects can be studied).
arguments for deception subjects like it
Arguments For Deception:Subjects Like It?
  • From H&O: Smith & Richardson (1983 observed that subjects in experiments w/ deception reported having enjoyed, & indeed having benefited from, experience more than those in experiments w/o deception.
  • See also Christensen (1988), Aitkenhead & Dordoy 1985; Sharpe et al. 1992).
  • Others have found opposite effect – Cook, Bean, Calder, Frey, Krovetz & Reisman 1970; Epstein, Suedfeld & Silverstein 1973; Allen 1983; Rubin 1985; Oliansky 1991; Fisher & Fyrberg 1994
empirical evidence on deception
Empirical Evidence on Deception
  • Experimental tests of trust games gives some evidence on effect of deception on subject behavior.
  • Suggest may accept being fooled once, but not twice (Dickhaut, Hubbard & McCabe 1995).
  • Recent results, Krupat & Garonzik (1994), suggest prior experience with deception affects subjects’ expectations – increases their suspicion (see also Epley & Huff 1998).
empirical evidence on deception102
Empirical Evidence on Deception
  • According to Krupat & Garonzik (1994), such suspicion is likely to introduce "considerable random noise" into their responses (p. 219).
  • Stang (1976) – percentage of suspicious participants (in conformity experiments) tracked closely increase in deception through 1960s.
ptt summary
PTT: Summary
  • Most PPT work is non formal, tests applications of theories & experimental results from psychology & social psychology to political science contexts, some deception.
  • Methodology differs from most FTT:
    • Less use of scripts
    • Process of individual decision making focused rather than choices in strategic situations.
    • Very little repetition of tasks
    • Financial incentives not tied to decision making