Meeting 11 Experimental methods I Metaphor in Language and Thought Ben Bergen

1. Meeting 11Experimental methods IMetaphor in Language and ThoughtBen Bergen

2. Today • HW2 returned • Course evaluation feedback • Experimental methods

3. Why experiment? • To answer questions about • online processing • unconscious cognitive processes • lexical representations • Independent of researchers’ intuitions and authority • Is the common currency of cognitive sciences • Forces researcher to consider mechanisms and to make discriminating predictions

4. Even if you don’t experiment • Need to be able to understand and critically assess experimental work • Principles we’ll discuss pertain more generally to most any aspect of language

5. Today • Developing an experiment • Defining a research question • Operationalizing it • Design details • Implementation

6. Defining a research question • Your research question is what you want to know. • Novel (don’t waste a year in the lab to save an hour in the library) • Interesting (because it bears on some broader issue) • Answerable • Justified by the current state of knowledge. E.g.: • A particular theory makes a prediction that another does not – which is right? • Previous empirical work on the topic fails to provided a convincing account, due to methodological or other difficulties. • You can reason out a prediction from first principles.

7. Defining a research question • A sample question:  • When people process metaphorical idioms (like He blew his stack), do they activate the source domain (in this case, heat)?  • This would be a good research question (in 1997): • Existing research didn’t answer it. • Answering it could tell us whether people access conceptual metaphors when they process metaphorical language, and by extension how they access meaning more broadly • It’s answerable, if you can measure the activation of source domains when people process language. • Previous theory argues that conceptual metaphors underlie language use

8. Operationalizing a question • To answer your research question • you have to transform it into a question about causation, namely, does X cause Y? • X is something that you can manipulate • Y is something that you can measure, like how long it takes someone to read a word or how often they pick one word or the other, etc. • Then you can ask whether manipulating X in a particular way causes differences in measurements of Y

9. Operationalizing a question • Independent variables and dependent variables • Thet thing you manipulate and that might affect the dependent variable is an independent variable or factor • Age, gender, handedness, etc. of the speaker • Native language of the experimental subject • Whether an image and a verb match • A potentially affected thing, which can be directly and reliably measured, is a dependent variable or dependent measure • Time to press a button. • Reading time for a word or sentence • Number of words remembered • Number of times a particular word is used

10. Operationalizing a question • Variables can be treated as continuous or categorical • Categorical variables have more than one level or condition • Whether the target word is related to the source domain or not • Whether the final phrase is a metaphorical idiom, literal paraphrase, or control • Continuous variables are measured on a scale • The reaction time (in miliseconds) to press a button

11. Operationalizing a question • Once you know your variables, you can rephrase your • research question as a pair of hypotheses. • Experimental hypothesis: the claim you want to test the likelihood of - that the independent variable measurably and significantly influences the dependent variable. • Null hypothesis: no relation between the variables.  • You are testing for evidence for the experimental hypothesis and against the null hypothesis. 

12. Design details • Any experiment is conducted with a number of • participants (the people taking part) • items (the things they're exposed to) • Participants • Shouldn't know the intent of the experiment (it's OK to lie, as long as it hurts no-one and you clear it up after). • Must agree to participate, and be informed of any risks and anonymity procedures http://www.hawaii.edu/irb/ • Must not differ in important ways across conditions (so you shouldn't have all old people in one condition and young people in another condition if that's not what you're testing.) • Must be representative of the population you’re interested in generalizing over

13. Design details • Items • Similar to each other as much as possible - pictures matched for size, brightness, etc.; words for frequency and length, etc., both within and across conditions. • You will sometimes also want to include filler items, which are not related to the experiment, and serve as a buffer, so the ppt doesn't catch on the manipulation

14. Design details • How many do I need of each? • You will usually want 5-20 observations per participant, per condition • So the number of items you need total will depend on the number of conditions you have • Depending on the size of the effect, you could need as few as 6 or as many as 100 subjects.

15. Design details • IV(s) can be manipulated either within or between ppts. • Within-participants: each ppt exposed to each condition of the IV. • Between-participants: each ppt exposed to only one condition of the relevant IV. • Advantages of within-participants design: • Controls for differences across participants • Gives statistical tests more power with fewer ppts • Advantages of between-participants design • No carryover effects, where exposure to one condition affects the other condition. • No need for repetition with small numbers of items • Sometimes required, when IV fixed for participants •  Many experiments have within- and between-ppts factors.

16. Implementation • An experiment can be high- or low-tech, depending on how finely you're manipulating stimuli and measuring responses. (If you need millisecond timing, or milimeter measurement, you have to use a computer.) • Some typically used tasks don't require a computer • Sorting objects or pictures into groups (categorization) • Putting objects in an order (representations of sequence) • Which are the same or which are different? (similarity) • Naming all the things you remember (memory) • Listing features, properties, parts, subtypes, etc. (features, etc. of concepts or categories) • Rating similarity, likelihood, etc. on a (Likert) scale from 1-7 (similarity, reasoning, etc.)

17. Implementation • Commonly used experiment design & presentation progs: • E-Prime: available only for Windows, not cheap, but well supported • Superlab: available for Windows and Mac, not cheap, but well supported • Psyscope: free and available for Mac, but unsupported. • DMDX: free, available for Windows, unsupported

18. Next steps • For more details on experimental methods and statistics, check out the readings for the next meeting