The Basics of Designing and Running an Economics Experiment

1. The Basics of Designing and Running an Economics Experiment Economics 328 Spring 2005

2. The Purposes of Running an Experiment • Test a Theory • Gather Empirical Regularities to Inform a Theory • Test Institutions

3. What Makes a Good Experiment? • Should an experiment replicate reality? • No! We have field studies for that. • Should an experiment replicate a formal model? • No! We have theory for that. • A good experiment tries to capture the most relevant features of reality in a simple, carefully controlled environment. Good experiments are usually designed to test specific hypotheses, sometimes derived from the implications of some economic theory and other times based on previous observations in either experiments or field data.

4. What is a Controlled Experiment? • As much as possible, the experiment controls all elements of the environment in which the experiment takes place. • The experimenter determines the rules – what choices are available to subjects, when decisions are made, and what the consequences of these decisions will be. • The experimenter controls subjects’ payoffs as a function of the actions they take. • The experimenter controls the information that is available to subjects.

5. What is a Controlled Experiment? • Between treatments, the experimenter only changes variables which are directly relevant to the hypothesis being tested, otherwise holding the environment fixed. • Control vs. Treatment Sessions • Avoid Confounds (e.g. Don’t change more than one thing at a time.) • Variables that cannot be directly controlled are typically controlled via randomization. Example: Many experiments are designed to test subjects’ attitudes towards fairness are affected by some treatment variable. To the extent that subjects enter the lab with differing attitudes about fairness, a true controlled experiment can’t be run. However, by randomly assigning subjects to treatments, we can eliminate subjects differing attitudes as a cause of differences between treatments. This relies on the law of large numbers, implying that a large sample may be necessary.

6. Designing an Experiment • Identify an interesting question or questions. These should be issues that are better addressed through a controlled experiment than through gathering field data. • Determine hypotheses you would like to test. Example: Increased payoffs lead to fewer mistakes in solving logic problems. Example: Face-to-face interaction will lead to greater concern with fairness. • Design a simple environment that allows you to test the hypotheses you are interested in. Remember, the more complicated the environment is, the more likely you are to lose control. An experiment is not meant to replicate reality. The reason we use controlled experiments is because reality is too complicated.

7. Brandts and Cooper:The Weak Link Game • The employee payoff function is given by the following equation: • Having all five players choose the same effort level is a Nash equilibrium. • The critical variable here is B, the bonus rate. Higher values of B give greater benefits to successful coordination. • Trying to move to a higher effort level is a risky strategy.

8. Brandts and Cooper:Research Questions • Can an increase in the bonus rate enable a firm to overcome coordination failure? • Does the magnitude of the bonus rate increase matter or is the simple fact of an increase effective as such? • If an increase in the bonus rate brings about improved coordination, can the bonus rate increase be revoked without affecting the improved outcome? • Does the length of time a firm has been underperforming affect the impact of the increase?

9. Designing an Experiment • Test your hypotheses by varying a small number of variables within the experiment. If at all possible, you should not vary more than one variable at a time. • Within vs. Between Subject Designs • Some important factors to remember . . . • Subjects get bored. • Subjects get confused. • Subjects learn as they gain experience within the experiment. You should anticipate changes in their behavior (order effects).

10. Design for Brandts and Cooper

11. Results for Brandts and Cooper

12. Results of Classroom Experiment

13. Results of Classroom Experiment

14. Designing an Experiment • Avoid the use of deceit. This is a very tempting trap, but one that is highly frowned upon by the profession. In particular, what is going to happen the next time you try to run an experiment? • Anonymity: Most experiments guarantee subjects anonymity. In other words, subjects are guaranteed that no other subjects (or indeed, nobody other than the researchers) will be able to ever identify their actions or payoffs. For some experiments, abandoning anonymity is an important part of the design. If so, subjects should know what information about them is be revealed publicly. Subjects should have the option of withdrawing if they do not want information about them revealed publicly.

15. Subjects • What population should you use as subjects? • Undergraduates are the easiest to get, but subjects with relevant experience are often more interesting to study. • How do you get subjects? • Advertisements can be placed on posters, electronic bulletin boards, or in newspapers. Spamming, while effective, is generally not a good way to make friends. • What should be in the advertisement? • It should include a very brief description, a summary of payoffs (average payoff is sufficient), the time necessary to complete the experiment, and contact information. (E-mail addresses work well. A website where people can sign up for the experiment works even better.) • The advertisement should stress the monetary payoffs. Remember, you want controlled experiments. This means that you want subjects who care about the monetary payoffs being offered.

17. Subjects • Avoid unintentional selection of the subject population. For example, if you post all of you ads in the weight room in Veale, your subject population will be skewed towards men. You should also avoid unintentional clustering (e.g. don’t sign up ten people from the same fraternity for a single session). • Have times and locations for sessions selected before posting any ads. Be certain to schedule potential subjects as soon as they respond to your ad. Be certain to get contact information from your subjects. Confirm that all subjects will attend the day before the session. Even with confirmation, you should anticipate that only about 80% of subjects will show up (CWRU students tend to be pretty good about showing up). • Keep careful track of all contacts with potential subjects. You should never use a subject twice for the same set of experiments. Most experimental groups keep a black list – subjects who don’t show up or disrupt sessions are undesirable.

18. Instructions • Most experiments include some form of consent form. For those of you who plan on publishing your research, this brings up the issue of the dread IRB . . . • The overarching goal of instructions is to make certain that the subjects understand the rules of the experiment. If subjects do not understand the rules, you have immediately lost control. • Instructions should be as clear and complete as possible. Any critical points should be repeated at least once. If in doubt, make the instructions too detailed. Nobody likes sitting through lengthy instructions, but remember your goal – the subjects need to understand the rules of the experiment. • Context: Many experiments like to frame their instructions in a generic context that does not have any real world connotations. In my opinion, this often generates confusion for experimental subjects, but that is a controversial point. All experimenters agree that you should avoid any loaded terms. • Example: In a Prisoner’s Dilemma Game, you wouldn’t want to label the two strategies as “Cooperate” and “Cheat” even though this is how game theorists often refer to them.

19. Instructions • Demand Effects: A common flaw of experimental instructions is that they suggest how the subjects ought to behave in a way that compromises the experimental design. This should be avoided scrupulously. Sometimes demand induced effects can be quite subtle. • Example: “The employees of a firm are always better off if all of them spend more hours on Activity A.” • Example: All of the examples in your instructions have exactly the same choices being made. • Payoff Quizzes: Payoff quizzes should be used to ensure that subjects understand the relationship between their choices, the choices of others, and their payoffs. Payoff quizzes also serve as a device for reinforcing important points from the instructions. Take care to avoid demand effects in writing your payoff quiz.

20. Instructions • Answering Questions: You should have subjects raise their hands and tell an experimenter any questions privately. (Use someone other than the person reading instructions.) This limits the chance of losing control by having one subject sharing his/her strategy with everyone in the session. Anything that you think is substantive, you should repeat aloud to the entire class. A common question is "Should I do blah, blah, blah . . ." or "What should I do . . ." Do not answer these questions directly, as this leads to demand effects. I always answer something like, "We don't know what you should do. The whole purpose of running the experiments is to see what you will do. You should do whatever seems best to you." Nobody has slugged me yet, although I imagine they have been tempted.

21. Actually Running the Experiment • Computers vs. Paper & Pencil Experiments • Run pilots in advance! Getting experimental subjects is hard work. You don't want to waste a room full of subjects because there is a problem with your procedures. Think everything out in advance and do some practice. • Setting Up the Room • Always set the room up in advance. Time is precious, so you don't want to waste time moving desks or handing out papers. • Separate people who arrive at the same time. You generally don't want subjects talking during the experiment, and people who arrive together are often friends. • Be certain that subjects can easily see you and any materials you might post in giving out instructions. Make it as hard as possible for them to see the materials being filled out by other subjects. Many established labs will have either dividers or sunken terminals. For our purposes, get a big room and spread people out.

22. Actually Running the Experiment • Keep careful records of subjects' arrivals to the lab. I usually keep a checklist. I usually will not allow any subject who is more than five minutes late participate in the session (10 if I'm desperate). I sometimes have done "bully recruiting," but this is generally best avoided. • You should generally have one person who runs the experiment (a "conductor) and a group of people who help out ("assistants"). The conductor is the front man. He/she reads the instructions, gives any general instructions, and oversees the experiment. The conductor need not have any particular authority in reality – the goal is to have someone with a loud voice who will seem credible to the subjects. The assistants do all the other myriad tasks involved in running an experiment. • Both the conductor and the assistant should be careful to avoid any demand effects. Do not lurk over the subjects while they are making decisions, do not rush them if they are thinking about a choice, and don't talk among yourselves unnecessarily during the session.

23. Actually Running the Experiment • You will want to have all forms made up in advance. In general, you will need the following sorts of forms. Make up a checklist before the experiment to make certain you don't forget anything. • Consent Forms (for publishable research) • Instructions • Payoff Tables: Make certain these are easy to read. Have these separate from the instructions so subjects can easily refer to them. • Record Sheets: You generally want a sheet that allows subjects to record all the information they have received, all the choices they have made, and all the payoffs they have received. This both allows them to know the history of their past choices and to confirm that you are paying them what they are owed. • Play Sheets: Sheets where subjects record their actions. • Receipts

24. Actually Running the Experiment • Paying Subjects: You should pay subjects in cash as privately as possible immediately after the end of the session. As much as possible, you should calculate the payoffs yourself rather than having the subjects do it. Be certain to get receipts!!! These need the subject's name, SS#, the date, the amount paid, and a signature. • Collecting Data: In designing your experiment, you should try to make it as easy as possible to figure out afterwards what happened. In particular, design the action sheets carefully and store them neatly as the experiment progresses. Buying lots of folders and labeling them in advance is a lot cheaper than spending days deciphering a mess. Do not rely on subjects' record sheets – these are often inaccurate.

25. Some Issues of Experimental Techniques • As much as possible, random number generation should be public and credible. For pencil and paper experiments, I try to use devices people will understand – rolling dice and flipping coins. Occasionally pre-generating the random numbers is desirable, although I try to avoid this. • Many experiments involve rematching subjects. If they are not supposed to be matched with the same person, you need to think out the matchings in advance to make certain that they won't be rematched. This is easy for 1 x 1 matchings, harder for group matchings. • Anonymity: For many experiments, we don't want subjects to know whom they are playing. For most experiments, using ID numbers rather than names will be good enough. For experiments with rematching, you may want to conceal the ID numbers. Some experimenters use more extreme techniques. For most cases, I think these are overkill.

26. Some Issues of Experimental Techniques • Risk Aversion: To limit the effects of risk aversion for experiments where it is a nuisance variable, many experiments use a "lottery technique." The value of this technique is dubious and it confuses the living daylights out of subjects. You should be sensitive to the possible effects of risk aversion on your results. • Feedback: Experimental results are quite sensitive to what feedback subjects are given. Any sort of feedback is acceptable, just think about its effects in advance and be certain that you aren't accidentally leaking additional information. Often times the feedback is an important component of the experimental design. • Asymmetric Information: Often times you want some information to only go to some subjects. Be certain that this doesn't slip over into deception. The other common problem is accidentally revealing information to people who aren't supposed to have it.