James W. Grice, Ph.D. Oklahoma State University Department of Psychology

1. Observation Oriented Modeling James W. Grice, Ph.D. Oklahoma State University Department of Psychology Presented to the faculty, students, and staff of New Mexico State University, Las Cruces, NM, March 8th, 2019.

2. What is Observation Oriented Modeling?

3. What is Observation Oriented Modeling? Positivism  Realism

4. What is Observation Oriented Modeling? Positivism  Realism Why does this matter? Costa, R. E., & Shimp, C. P. (2011). Methods courses and texts in psychology: "Textbook Science" and "Tourist Brochures." Journal of Theoretical and Philosophical Psychology, 31(1), 25-43. “It matters because a commitment to a philosophy of science generally colors what one believes science is, and that in turn impacts what one believes is good science, what is appropriate scientific methodology, and what is the proper relation between science, truth, and society. Therefore, it affects the process of doing science…” (p. 35)

5. What is Observation Oriented Modeling? • Marvin Dunnette (1966): Fads, fashions, and folderol in psychology, American Psychologist, 21(4), 343-352. • David Lykken (1991): What’s wrong with psychology, anyway? • Thinking clearly about psychology. Minneapolis: University of Minnesota Press. • Psychology’s research tradition is fundamentally flawed • Majority of published psychological studies are non-replicable • Psychology can claim very little in the way of cumulative knowledge • Brad Woods (2011). What’s still wrong with psychology, anyway? • Unpublished Thesis, University of Canterbury, NZ. • Ferguson, C. J. (2015). “Everybody knows psychology is not a real science”: Public perceptions of psychology and how we can improve our relationship with policymakers, the scientific community, and the general public. American Psychologist, 70(6), 527-543. • Replication crisis • Null Hypothesis Significance Testing problems (p-values) • Publication bias • Questionable Research Practices

6. What is Observation Oriented Modeling? Unrealistic Measurement Assumptions toTaking a Data Set “as is”

7. Stress-Vulnerability Signatures Study Primary Question: Individual triggers for stress?

8. Stress-Vulnerability Signatures Study Hierarchical Linear Modeling Level 1: Yij = β0j + β1jXij + εij Level 2: β0j = γ00 + γ01Wj+μoj β1j = γ10 + μ1j Assumptions: Linearity Normality Homoscedasticity Independence Quantitative measurement; here, 0 – 10 scale is a continuous measure of stress, anxiety, incompetence, etc.

9. Stress-Vulnerability Signatures Study Feeling Excluded Feeling Inferior

10. Stress-Vulnerability Signatures Study Hierarchical Linear Modeling Jack1 = 10, Jack2 = 5 Is Jack twice as stressed on Day 1 compared to Day 2? Jack1 = 10, Jack2 = 4 Jack3 = 4, Jack4 = 1 (10 – 4) / (4 – 1) = 2 Is Jack’s change in stress from Day 1 to Day 2 twice that of Day 3 to Day 4? Level 1: Yij = β0j + β1jXij + εij Level 2: β0j = γ00 + γ01Wj+μoj β1j = γ10 + μ1j Assumptions: Linearity Normality Homoscedasticity Independence Quantitative measurement; here, 0 – 10 scale is a continuous measure of stress, anxiety, incompetence, etc. “There is no evidence that the attributes that psychometricians aspire to measure (such as abilities, attitudes and personality traits) are quantitative…All the evidence is that these attributes are merely ordinal…” Michell, J. (2011). Qualitative research meets the ghost of Pythagoras. Theory & Psychology, 21(2), 241-259. Quote is from page 245, italics original.

11. Stress-Vulnerability Signatures Study “In conclusion, as a matter of fact the unsatisfactory situation endures; that is, so far, no interval or ratio scales have been established in psychology, neither by conjoint measurement nor by any other means. Hence, if one does not want to abstain from using methods of data analysis which rely on the assumption that the measurement problem has been successfully solved (e.g., structural equation modeling, Bollen, 1989; or multiple regression/correlation analysis, Cohen & Cohen, 1975) the question of measurability is to be categorized as urgent. As has been emphasized by Paul Barrett (2008), the consequence of ignoring the issue is scientific stagnation (see also Barrett, 2018). Apparently, the acuteness of the problem is still underestimated.” Trendler, Günter (2018). Conjoint measurement undone. Theory & Psychology, 1-29. Quote is from page 2.

12. Stress-Vulnerability Signatures Study Summarizing David A. Freedman’s classic paper, Statistical Models and Shoe Leather, Mason writes: “Simple [statistical/data analytic] tools should be used extensively. More complex tools should be used rarely, if at all. Thus, we should be doing more graphical analyses and computing fewer regressions, correlations, survival models, structural equation models, and so on.” Freedman, D. A. (1991). Statistical models and shoe leather. Sociological Methodology, 21, 291-313. Mason, W. M. (1991). Freedman is right as far as he goes, but there is more and it’s worse. Statisticians could help. Sociological Methodology, 21, 337-351. Quote is from page 338.

13. Stress-Vulnerability Signatures Study Efficient Cause Analysis Percent Correct Classifications (PCC) = 96.00%

14. Stress-Vulnerability Signatures Study Feeling Excluded Feeling Inferior

15. What is Observation Oriented Modeling? Unrealistic Measurement Assumptions toTaking a Data Set “as is”

16. What is Observation Oriented Modeling? Inference to Population ParameterstoInference to Best Explanation

17. Affective Realism Study

18. Affective Realism Study

19. Affective Realism Study μPositive μNeutral Mean Rating (1 – 5 scale) μNegative Positive Neutral Negative Condition Alternative Hypothesis, HA : > >

20. Affective Realism Study μPositive μNeutral μNegative Mean Rating (1 – 5 scale) Positive Neutral Negative Condition Null Hypothesis, H0 : = =

21. Affective Realism Study N = 45 Northwestern Undergraduate Students 180 Trials of paired faces Dependent Variable = Mean rating (1 to 5), per person, for 180 trials Independent Variable = Three subliminal stimuli (positive, neutral, negative) Repeated Measures ANOVA F(2,42) = 9.72, p < .001, η2 = .32 (CI.95: .08, .48) Two-sided Bayesian repeated ANOVA with a default Cauchy prior width of r = .707 Bayes factor (BF10) = 62.9 “…indicating that the observed data are 62.9 times more likely under the alternative hypothesis (that suppressed affective information will influence perception) than under the null hypothesis. A Bayes factor of 62.9 is considered very strong evidence in favor of our hypothesis.” (p. 499) Follow-up pairwise ANOVAs HA: > p = .04 HA : > p = .03 HA : > p = .001

22. Three Types of Inference = population mean = ?, N = ? = sample mean = 3.02 H0 : = = p < .05 … Inference to Population Parameters Not Person-Centered, nor Explanatory n = 45 …

23. Three Types of Inference H0 : = = • Assumptions • Random sampling • Normal population distributions • Homogeneity of population variances • Continuous dependent variable • Independence of observations • Ho is true “Random sampling from a large, well-defined population or universe is a formal requirement for the usual interpretation of all commonly used parametric and nonparametric statistical tests.” (Dugard, 2014) “Inferential statistics are used in reasoning from a sample to the population … [convenience samples], unlike random samples, don’t provide a sound basis for determining the properties of populations.” (Kirk, 1999).

24. Three Types of Inference Participants and the cytoarchitecture of their individual brains.

25. Three Types of Inference Cytoarchitecture S & P Cytoarchitecture S & P Abductive Inference Person-Centered and Explanatory

26. Three Types of Inference Jill Jack Mean Rating (1 – 5 scale) Joe Positive Neutral Negative Condition

27. Three Types of Inference OOM Results Cytoarchitecture S & P Abductive Inference Person-Centered and Explanatory All 45 students: PCC = 24.44% 5 (11.11%) students’ responses were exactly opposite of the predicted pattern.

28. Three Types of Inference Inference to Best Explanation Compare Competing Causal Models (Best) 2. Evaluate Data Pattern Against Chance (Low Hanging Fruit)

29. Randomization Tests n = 10; PCC = 80.00%

30. Randomization Tests Why are the data patterned in this way? Charles S. Peirce Inference to Best Explanation: I1: The patterned triplets of observations are best explained by the structures and processes outlined by the cytoarchitecture model of affective realism. I2: The patterned triplets of observations are best explained as arbitrarily connected (i.e., connected “by chance”)

31. Randomization Tests PCC = 20.00%

32. Randomization Tests PCC = 30.00%

33. Randomization Tests 0.00 : [ 81015]*************************** 10.00 : [161605]****************************************************** 20.00 : [145342]************************************************ 30.00 : [ 76970]************************** 40.00 : [ 27230]********* 50.00 : [ 6591]** 60.00 : [ 1126]* 70.00 : [ 113]* 80.00 : [ 8]* 90.00 : [ 0] 100.00 : [ 0] Total Number of Observations : 500000 Only 8 of the 500,000 randomized versions of the data yielded a PCC index of at least 80.00%. The c-value is therefore reported as: c-value = 1.6 x 10-5(or .000016)

34. Randomization Tests “All that a randomization test tells us is that a certain pattern in the data is or is not likely to have arisen by chance” (Manly, 1991, p. 2). A randomization test is the first step “to exclude the hypothesis of chance” (Winch & Campbell, 1969, p. 143) in route to a theoretically meaningful inference.

35. Randomization Tests Why are the data patterned in this way? Charles S. Peirce Inference to Best Explanation: I1: The patterned triplets of observations (n = 10; PCC = 80%) are best explained by the structures and processes outlined by the cytoarchitecture model of affective realism. I2: The patterned triplets of observations are best explained as arbitrarily connected [i.e., connected “by chance” (c-value = .000016)]

36. Three Types of Inference OOM Results Cytoarchitecture S & P Inference to Best ExplanationPerson-Centered and Explanatory All 45 students: PCC = 24.44% 5 (11.11%) students’ responses were exactly opposite of the predicted pattern. c-value = .10

37. Randomization Tests Advantages : Very few or no necessary assumptions“…randomization tests do not need the unrealistic assumptions of classical tests (including random sampling).” (Dugard, 2014, p. 68) Can be used with small samples and single-case designs(Todman & Dugard, 2001) Can be used with experimental and correlational designs(see Manly, 1991, for examples) Intuitive and easy to understand or teach Can be used with any statistic…all you have to do is repeatedly randomize the data and create the statistic’s distribution. Match behavior of researchers, most of whom rarely draw random samples(Ludbrook & Dudley, 1998)

38. What is Observation Oriented Modeling? Variable-Based to Integrated Models Assumption-laden p-valuestoAssumption-free Randomization Tests

39. Memory Study Fitness-Related Processing will enhance memory performance. This is an evolutionary adaptation of humans.

40. Memory Study • Students • Imagine surviving in a foreign grasslands • While imagining, they view 8 words (e.g., lawyer, basketball) and rate their relevance to the scenario.Totally irrelevant 1 2 3 4 5 Extremely relevant • Imagine being on a tropical vacation • While imagining, they view 8 different words (e.g., sword, pope) and rate their relevance to the scenario.Totally irrelevant 1 2 3 4 5 Extremely relevant • Repeat for SVSV (or VSVS) data • Number short-term memory task • Surprise recall of words • Expectation: More words will be recalled from survival scenario than from vacation scenario.

41. Memory Study As part of the Reproducibility Project in Psychology, the replication team found that, in line with the original study, survival processing produced a significant recall advantage, F(1, 37) = 8.08, MSE = .021, p = .007. We, too, found such “statistically significant” differences with a novel sample (n = 99) of OSU students. Condition (S vs. V) # RecallWords

43. Memory Study Abductive Inference as prelude to Inference to Best Explanation: I1: Recalling words in this task is best explained by fitness-related processing; specifically, the pattern “survival > vacation” is best explained by this mechanism. Importantly, Nairne and Pandeirada (2016, p. 500) suggest an alternative explanation (inference) of such results: “If the relevance of the to-be-rated word to the survival scenario is immediately obvious …then there is little need to engage in much fitness-relevant processing.” Inference to Best Explanation: I1: Recalling words in this task is best explained by fitness-related processing; specifically, the pattern “survival > vacation” is best explained by this mechanismI2: Recalling words in this task is best explained by a well known mnemonic process; specifically, the pattern “relevant words > non-relevant words” is best explained by this mechanism.

44. Pattern Analysis : Multiple Crossed Orderings

45. Competing Causal Models of Memory Recall Percentage of Individuals for which Model #2 provided a more accurate account of responses compared to Model #1 = 70.71% (PCC), c-value < .001

46. Memory Study Inference to Best Explanation: I1: Recalling words in this task is best explained by fitness-related processing; specifically, the pattern “survival > vacation” is best explained by this mechanismI2: Recalling words in this task is best explained by a well known mnemonic process; specifically, the pattern “relevant words > non-relevant words” is best explained by this mechanism.

47. Memory Study …but, there’s more: Original “Survival > Vacation” effect was due to a confound Fortuitous pairing of words/scenarios Hammer, shirt, sword, salmon, truck, and horse were matched with the survival scenario in one of the conditions, and this pairing drove the effect that led to the conclusion in support of fitness-related processing. OOM provides the tools for improving this research!

48. Thank You! http://www.idiogrid.com/OOM