Techniques expérimentelles 2

1. Techniques expérimentelles 2 Barbara Hemforth Most of thisisstolenfrom a lecture by Chuck Clifton http://webcache.googleusercontent.com/search?q=cache:nPJ6GhwZ2NkJ:people.umass.edu/cec/Experimental%2520Design%2520for%2520Linguists.ppt+Clifton+experiments+linguists&cd=5&hl=fr&ct=clnk&gl=fr&client=safari&source=www.google.fr

2. II: How to do experiments. Part 1, General design principles • Dictum 1: Formulate your question clearly • Dictum 2: Keep everything constant that you don’t want to vary • Dictum 3: Know how to deal with unavoidable extraneous variability • Dictum 4: Have enough power in your experiment • Dictum 5: Pay attention to your data, not just your statistical tests C. Clifton Jr

3. Dictum 1: Formulate your question clearly • Independent variable: variation controlled be experimenter, not by what subject does • Dependent variable: variation observed in subject’s behavior, perhaps dependent on IV • Operationalization of variables C. Clifton Jr

4. Dictum 2: Try to keep everything constant except what you want to vary • Try to hold extraneous variables constant through norms, pretests, corpora… • When you can’t hold them constant, make sure they are not associated (confounded) with your IV

5. What happens when there is unavoidable variation? • Subdictum B: When in doubt, randomize • Random assignment of subjects to conditions • Questionnaire: order of presentation of items? • Single randomization: problems • Different randomization for each subject • Constrained randomizations • Equate confounds by balancing and counterbalancing • Alternative to random assignment of subject to conditions: match squads of subjects

6. Counterbalancing of materials • Counterbalancing • Ensure that each item is tested equally often in each condition. • Ensure that each subject receives an equal number of items in each condition. • Why is it necessary? • Since items and subjects may differ in ways that affect your DV, you can’t have some items (or subjects) contribute more to one level of your IV than another level.

7. Sometimes you don’t have to counterbalance • If you can test each subject on each item in each condition, life is sweet • E.g., Ganong effect (identification of consonant in context) • Vary VOT in 8 5-ms steps • /dais/ - /tais/ • /daip/ - /taip/ • Classify initial segment as /d/ or /t/ • Present each of the 80 items to each subject 10 times • Ganong effect: biased toward /t/ in “type,” /d/ in “dice” Connine, C. M., & Clifton, C., Jr. (1987). Interactive use of information in speech perception. Journal of Experimental Psychology: Human Perception and Performance, 13, 291-299.

8. If you have to counterbalance… • Simple example • Questionnaire, 2 conditions, N items • Need 2 versions, each with N items, N/2 in condition 1, remaining half in condition 2 • Versions 1 and 2, opposite assignment of items to conditions • More general version • M conditions, need some multiple of M items, and need M different versions • Embarrassing if you have 15 items, 4 conditions… • That means that some subjects contributed more to some conditions than others did; bad, if there are true differences among subjects

9. Counterbalancing things besides items • Order of testing • Don’t test all Ss in one condition, then the next condition… • At least, cycle through all combinations of conditions (all lists) before testing a second subject with the same list • Fancier, latin square • Avoid minor confound if always test cond 1 before cond 2 etc. • N x n square, sequence x squad, containing condition numbers, such that each condition occurs once in each column, each order • Location of testing • E.g., 2 experiment stations

10. Latin Squares (Euler, 1773) Latin square of order 2 Latin square of order 3 ab x   y   z b   a z   x   y y   z   x • A latin square of order n is an n by n array of n symbols in which every symbol occurs exactly once in each row and column of the array.

11. Variance in an experiment • Systematic variance: variability due to manipulation of IV and other variables you can identify • Random variance: variability whose origin you’re ignorant of • Point of inferential statistics: is there really variability associated with IV, on top of other variability? • Is there a signal in the noise?

12. Best way to deal with extraneous variability: Minimize it! • Keep everything constant • Reduce experimental noise • See the signal easier • Keep environment, instructions, distractions, experimenter, response manipulanda, etc. constant • Pretest subjects and select homogeneous ones, if that suits your purposes

13. One way to minimize extraneous variance: Within-subject designs • Subjects differ • …a lot, in some measures, eg. Reading speed, reaction time • Present all levels of your IV to each subject • Assume the subject effect is a constant across all the levels. • Differences among conditions thus abstracted from subject differences • Counterbalancing necessary • Test each item in each condition for an equal number of subjects. • Worry about experience changing what your subject did • E.g., will reading an unreduced relative clause (The horse that was raced past the barn fell) affect reading of a reduced relative clause sentence?

14. Dictum 4: Have enough power to overcome extraneous variability • Add more data! • Minimizes noise component of differences among condition means • Law of large numbers • The larger the sample size, the more probable it is that the sample mean comes arbitrarily close to the population mean • If you’re (almost) looking at population means, any differences have to be real – not sampling error

15. Dictum 5: Pay attention to your data, not just your statistical tests • Look at your data, graph them, try to make sense out of them • Don’t just look for p < .05! • Examine confidence intervals • Look at your data distributions • Stem and leaf graphs • By subjects…

16. Magnitude estimation:Steven’s Power Law http://www.cis.rit.edu/people/faculty/montag/vandplite/pages/chap_6/ch6p10.html

17. Steven’s Power Law http://www.cis.rit.edu/people/faculty/montag/vandplite/pages/chap_6/ch6p10.html

18. Steven’s Power Law http://www.cis.rit.edu/people/faculty/montag/vandplite/pages/chap_6/ch6p10.html

19. Magnitude estimation: an example Which man did you wonder when to meet? Assign an arbitrary number to that item, greater than zero. Now, for each of the following items, assign a number. If the item is better than the first one, use a larger number; if it’s worse, smaller. Make the item proportional to how much better or worse the item is than the original – if twice as good, make the number 2x the start; if 1/3 as good, make the number 1/3 as big as the start.

20. Magnitude estimation : an example • Which man did you wonder when to meet? • Assign an arbitrary number, greater than 0, to this first item. • Now, for each successive item, assign a number – bigger if the item is better, smaller if worse, and proportional – if the item is 2x as good, make the number 2x the original; if ¼ as good, make the number ¼ as big as the original. • Which book would you recommend reading? • When do you know the man whom Mary invited? • This is a paper that we need someone who understands. • With which pen do you wonder when to write. • Who did Bill buy the car to please? Bard, E. G., Robertson, D., & Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 72.