Model Selection and Inference: Motivation, Mechanics, and Interpretation

1. Model Selection and Inference: Motivation, Mechanics, and Interpretation Gail Olson and Dan Rosenberg Department of Fisheries and Wildlife Oregon State University www.oregonstate.edu/~rosenbed/workshop.htm

2. Goal of Workshop • Provide motivation for a conceptually simple approach • for the analysis of data using multiple models • emphasizing an a priori approach • Provide the mechanics of how to use AIC • Guidance on how to interpret results from an AIC approach • Discuss how this may benefit your research

3. Starters We assume: The research started with an intriguing and important question AND You used a proper experimental or probability-based sampling design Analytical strategies can not account for the failure of these points The research started with an intriguing and important question You used a proper experimental or probability-based sampling design

4. reliable Goal of Research in Management of Natural Resources • understand nature and how it reacts to perturbations • make predictions based on inferences from analysis of empirical data

5. Why models? Steps in Making Reliable Inferences • Inference from Sample to the Population • Identify and understand patterns and mechanisms • Statistical models to aid detection and interpretation Pr(use) distance “All models are wrong, but some are useful” Box (1976)

6. Statistical models as expression of specific hypotheses What is Meant by “Model”? • 1. Theory: A hypothesis that has survived repeated efforts to falsify it • Hypothesis: a story about how the world works • Model: an abstraction or simplification of the real world; models as tools for the evaluation of hypotheses • Statistical models separate noise from information inherent in “data” This is particularly important in the model selection framework; recognition that there is not necessarily a single model appropriate for inference

7. A trivial straw man • alpha level is arbitrary • Emphasis on the test itself; usually not informative • Probability of use is unrelated to distance from a nest, • habitat type, and landscape context • Null (H0):  d = 0;  hab1 =  hab2=  hab3;  patch size = 0 • Alternative (HA ): 0 > d > 0; etc…… • Reject(H0) if test statistic is such that p  0.05 • That is, if the prob that the data arose from the null • is exactly  0.05, reject H0in favor of HA • “Habitat selection was not significantly different • among crop types (P < 0.05).” Single vs Multiple Models Traditional Hypothesis Testing (Single Model)

8. Traditional Hypothesis Testing (Single Model) All we typically learn is that the sample sizes were not large enough to detect differences

9. A Multiple Model Approach • Avoid “pet” hypothesis • All models equally likely in their selection or weight • Simultaneously comparing and ranking models • Emphasis on direction, magnitude, and precision of effects • Estimates can be based on multiple models Single vs Multiple Models • Probability of use is : • unrelated to distance from a nest • related linearly • related exponentially All hypotheses receive equal initial weight in evaluation, and all models can be used in inference so one does not have to select a single model

10. Emphasis on an a priori Model Set

11. These all become “candidate models” • formulation of a useful set of a priori models • selection and weighting models for ranking hypotheses • and parameter estimation Goal: Hypotheses Expressed as Statistical Models • A Global Model • has many parameters representing plausible effects and the state of the science, as well as relevant study design issues; most complex model of set • Subsets • can be considered special cases of the global model; fewer parameters, not necessarily nested; always of same response variable and estimated from the same set of data

12. Simplicity and Clarity as Goals Developing an a priori Model Set • Have the question crystal-clear • Bring in your (team’s) understanding of the problem • Incorporate past research via literature review 4. Understand the expectation of the process based on theory and include this expectation in your model set • 5. Include models of opposing views • 6. Should be subjective– bring in various views and thoughts • 7. Avoid all possibilities “just because you can” • Number of parameters must be considered in terms of sample size 9. Number of models should be a balance between small number of biologically plausible models and not excluding potentially important models

13. A Model of Habitat Selection N Per unit area, Pr (use) = f(dist. to focal site) + barriers + attractants

14. Hypotheses and their Rationale A. Hypotheses related to distance effects Pr (Use) Distance from the Nest

15. B. Hypotheses related to Crop Type • cover type by dominant species • cover type by structure C. Hypotheses related to landscape characteristics • patch size • distance to perennial crop • dominant type within home-range Hypotheses and their Rationale

16. The Set of Candidate Models • Global Model: The most complex model • Pr(use)= distance (polynomial), crop types, patch type, • distance to perennial crop, dominant in home range • Model Subsets: Includes one or more parameters • distance (linear) • distance (log) • distance (polynomial) • Crop-Only models • includes parameter for each crop type • Crop types combined into structure classes • Best distance model + crop parameters • Best Distance model + structure parameters • No effects model • Best distance + cover or crop model + patch type • Etc.

17. Model Sel. Criteria Crop Cover Dist (L) Dist (P) D(l)Crop D(P)Crop D(l)Cover D(P)Cover

18. Conformity of Burrowing Owl Space-Use Patterns to the Central-Place Model Large individual (and/or sampling) variation Percent Locations Agriculture Fragmented Distance (km) from Nest

19. Summary: Motivation for an a priori Model Selection Approach Statistical models to separate pattern from noise Single vs. multiple model approaches Insignificance of Statistical Significance Testing (Johnson 1999) Emphasis on parameter estimation and uncertainty Ranking and evaluating competing hypotheses Inference from multiple models often difficult to identify the best model

20. Akaike’s Information Criterion (AIC) • Metric to rank and compare models • Hirotugu Akaike (1973) “An Information Criterion” • Simple metric with DEEP theory Boltzmann’s entropy – Physics Kullback-Leibler discrepancy – Information theory Maximum Likelihood Theory - Statistics

21. Kullback-Leibler Discrepancy

22. Maximum Likelihood (ML) • Good statistical properties Unbiased Minimum variance • Links models, parameters, data • L (parameters | model, data) • Usually expressed as a log value: log (L (q|g(y),y)) • Aim is to maximize the log value

23. ML Example • Binomial model L (p | binomial, y) • Log (L (p | bin, y)) For n=11 and y=7

24. Model over-fitting

25. Principle of Parsimony

26. ^ -2logL = -2log(L (q| y)) (Model fit) k = number of parameters “Penalty” AICBasics AIC = -2logL + 2k

27. AICc for small sample sizes • Less biased • Use when n/k < 40 • Better, use all the time!

28. Model Selection • Compute AICc for each model • Ranklowest to highest • Lowest AICc = “best” model • Example: Northern Spotted Owl Survival Analysis Effects of Seasonal Climate covariates (Precipitation and Temperature)

29. Model ranking by AICc

30. DAICc DAICc = AICc(model) – AICc(min) Compare model relative to “best” model Rules of Thumb (B&A): 0-2 = Competing, substantial support 4-7 = Less supported 10+ = Essentially no support

31. Relative rankings

32. Akaike weights • Relative likelihood of each model • Specific to model set (Swi=1)

33. Model weights S

34. Model weights

35. Fun things to do with weights • Evidence ratios Compare one model to another • Confidence sets What models are more likely? • Importance values What variables are most important?

36. Evidence Ratios Compare best model (Pen) with “no climate model”: Wpen = 0.3318 , Wno climate = 0.1040 ER = 0.3318/0.1040 = 3.19 Pen model ~ 3X more likely than no climate model

37. 95% Confidence Set

38. Importance values • Cement Hardening Example (B&A) • Time to hardening based (y) on composition of 4 different ingredients (xi) • Regression: y = b0+b1(x1)+b2(x2)+b3(X3)+b4(x4)

39. AIC in regression analyses • Number of parameters: k = number of variables (xi) + intercept (if used) + error variance (s2) • AIC may be calculated from (s2) as: AIC = nlog (s2) + 2k ^ ^ ^

40. Multi-model inferenceModel Averaging • Incorporates model selection uncertainty • Used for parameter estimation Directly estimated or not E.g. Regression coefficients, predicted values

41. Pitfalls to avoid • Use same data set for all models Caution: missing values • Transform X’s but not Y • Number of parameters known? “hidden” parameters “lost” parameters Bottom line: Know what you are doing!

42. Interpreting Results Some issues: Models differing by 1 parameter Model ambiguity Null model best Model redundancy

44. Model Ambiguity NSO Productivity Modeled as function of Habitat covariates