New Directions

1. New Directions Integrated framework Full Bayesian approach that applies to internal and external validity (Tenglong Li) Value added: for a teacher or school rated as ineffective, how many students would you have to replace with average students to achieve an effective rating? (Qinyun Lin) Application to Comparative Interrupted Time Series (CITS) – Spiro Maroulis Mediation (Qinyun Lin) Non-linear (e.g., logistic) What is a null hypothesis case? P(success) is independent of predictor multilevel regression Can be thought of as weighted least squares, assumes weights not related to replacement of cases Propensity score matching How many bad matches would you have to have to invalidate inference (Rubin liked this idea)

2. Integrated Framework In case replacement, πi represents the probability of a case being replaced. In the omitted variables case πi is constant across all i. The omitted variable = f(yideal -yobserved ) where f is any linear function In the weighted case, 0<πi< 1, varying across cases.

3. Presented at AEFP, Portland Oregon, March 2018

4. Interaction and replacement cases predicting test score Replacement cases

5. Difference-in-DifferenceCounterfactual Assumption • Can swing red dashed line and see who much swing we need to invalidate inference • It can swing because of 1) what happens at Ypre 2 or 2) between Ypost and Y pre 2 • Sharpens two different confounding conversations • Can compare % bias to invalidate with the std err of pretreatment slopes • SPIRO: try all of this on the JAMA suicide attempt example • What if we treated the trend covariate as a covariate for which there was “balance” – what does it mean when we do that – it’s a comparision to a theoretical experiments that has many, many “states” • We are in world where rvt is low and rvy is high (v is the pre-trend) . Outcome . . Tpre 2 . . Cpost Tpre 1 Cpre 2 . Cpre 1 Time Pre-treatment period Treatment occurs, t Post-treatment period

6. Diffence-in-DifferenceCounterfactual Assumption(Spiro Maroulis) . Outcome Tpost D-i-D Treatment effect . Counterfactual: What would have happened to treatment group if they had not actually received the treatment . Tpre 2 . . Cpost Tpre 1 Cpre 2 . Cpre 1 Time Pre-treatment period Treatment occurs, t Post-treatment period

7. Reflection What part if most confusing to you? Why? More than one interpretation? Talk with one other, share Find new partner and problems and solutions

8. Robustness and Replicability

9. Original study effect size versus replication effect size (correlation coefficients). data https://osf.io/ezcuj/wiki/home/ Open Science Collaboration Science 2015;349:aac4716 Published by AAAS

10. THE BAYESIAN PARADIGM OF ROBUSTNESS INDICES OF CAUSAL INFERENCESDoctoral dissertation by Tenglong Li, Michigan State February 2018 ABSTRACT The validity of a causal inference hinges on a research design with both strong internal validity and strong external validity (Shadish et al. 2002). Unfortunately, such research is rare so that causality is typically inferred through a small-scale randomized experiment or a large-scale observational study (Schneider et al. 2007). In light of this gap, the robustness indices of causal inferences have been proposed by Frank et al. (2013) to measure the robustness of causal inference by quantifying the proportion of the observed sample that needs to be replaced with unfavorable unobserved cases. Drawing on the Bayesian discussion in Frank & Min (2007), this dissertation purposes developing the Bayesian framework of the robustness indices of causal inferences for causal research with either limited internal validity or limited external validity. This dissertation has two chapters: The first chapter lays the foundation of the Bayesian paradigm of robustness indices by formally defining prior as distribution built on an unobserved sample. For a particular family of prior and likelihood distributions, the posterior can be interpreted as distribution built on an ideal sample. The Bayesian paradigm of robustness indices of causal inferences focuses on the relationship between the posterior probability of invalidating an inference and the unobserved sample statistics and the central task is to locate the threshold of an unobserved sample statistics with regard to a given value of the posterior probability of invalidating an inference. Considering the first chapter targets the simple group-mean-difference estimator only, the second chapter extends the Bayesian paradigm of robustness indices to regression models. This dissertation promotes the scientific discourse of causality and critical thinking by linking the probability of invalidating an inference to detailed thought experiments characterized by the thresholds of sufficient statistics pertaining to an unobserved sample.

11. 'Crosnoe, Robert' <crosnoe@austin.utexas.edu>; rwerum2@unl.edu; Muller, Chandra L <cmuller@austin.utexas.edu>; bollen@unc.edu; simon.cheng@uconn.edu; drew@drewdimmery.com; bradley.blaylock@okstate.edu; fabiog@ntu.edu.sg; shevlin@u.washington.edu; ben.b.hansen@umich.edu; chosman@umich.edu; paul.holland200@att.net; efjuncai@cityu.edu.hk; fbrykho@cityu.edu.hk; mjs3@princeton.edu; gabriela.olivares@uni.edu; klugman@temple.edu; mgaddis@live.unc.edu; Arthur.Kraft.1@city.ac.uk; HuaiZhang@ntu.edu.sg; jeeseonkim@wisc.edu; michael.weiss@mdrc.org; mgangl@ssc.wisc.edu; dharding@umich.edu; dlarcker@stanford.edu; trusticus@london.edu; tad61@columbia.edu; mfeng@katz.pitt.edu; chanli@katz.pitt.edu; gpainter@usc.edu; mcorreia@london.edu; jcore@mit.edu; ksmith1@unl.edu; ssbsup@okstate.edu; Spiro Maroulis <Spiro.Maroulis@asu.edu>; 'Ben Kelcey' <ben.kelcey@gmail.com>; 'Minh Duong' <mdthetahat@gmail.com>; mseltzer@ucla.edu; cwinship@wjh.harvard.edu; 'James Moody' <jmoody77@soc.duke.edu>; 'William Carbonaro' <William.Carbonaro.1@nd.edu>; 'Mark Berends' <Mark.Berends.3@nd.edu>; Min Sun <misun@uw.edu>; Richard.Taffler@wbs.ac.uk; machteld.vandecandelaere@ppw.kuleuven.be; annowens@usc.edu; keongkim@cityu.edu.hk; smithas@rhsmith.umd.edu; jwhitaker@richmond.edu; pkr@pkr.in; eminor1@nl.edu; Kaitlin Torphy <torphyka@gmail.com>; I-Chien Chen <ichiench@msu.edu>; loyunjia@msu.edu; liuyqphilly@gmail.com; Sarah Galey <sarahgaley@gmail.com>; Jihyun Kim <toveave@gmail.com>; qinyun Lin <linqinyun1001@gmail.com>; Aaron Zimmerman <aaron.samuel.zimmerman@gmail.com>; Kim Jansen <jansenki@msu.edu>; zixi chen <zixichen68@gmail.com>; litenglo@msu.edu; John Lane <lanejoh3@msu.edu>; Guan K. Saw <sawguan@msu.edu>; kmartin@msu.edu; Ran Xu <ranxu@msu.edu>; Christopher Lee <cdeanlee@umich.edu>; Rachel White <whitera3@msu.edu>; Angelo Joseph Garcia <angelo.giuseppe63@gmail.com>; Tingqiao Chen <chentin4@msu.edu>; Yincheng Ye <yycheng@vt.edu>; r-korajczyk@kellogg.northwestern.edu; R.H.Cuijpers@tue.nl; liuyqphilly@gmail.com; ranxu@msu.edu; sarahgaley@gmail.com; toveave@gmail.com; chentin4@msu.edu; linqinyun1001@gmail.com; aaron.samuel.zimmerman@gmail.com; torphyka@gmail.com; loyunjia@msu.edu; ichiench@msu.edu; croninge@umd.edu; 刘雨甲 <yujia.liu@bnu.edu.cn>; djklasik@umd.edu; Ruth N. Lopez Turley <turley@rice.edu>; David Williamson Shaffer <david.williamson.shaffer@gmail.com>; dkaplan@education.wisc.edu; Eric Camburn <ecamburn@education.wisc.edu>; Geoffrey Borman <gborman@education.wisc.edu>; georgek@uic.edu; stephen.morgan@jhu.edu; Cavanagh, Shannon E <scavanagh@austin.utexas.edu>; Dedrick, Robert <Dedrick@usf.edu>; John Lockwood <jrlockwood@ets.org>; Lou Mariano <Lou_Mariano@rand.org>; Joshua Cowen <jcowen@msu.edu>; pay2n@virginia.edu; T Gmail <tdietzvt@gmail.com>; Yu Xie <yuxie@umich.edu>; smithas@rhsmith.umd.edu; Konstantopoulos, Spyros <spyros@msu.edu>; Kimberly S. Maier <kmaier@msu.edu>; Schmidt, William <bschmidt@msu.edu>; Houang, Richard <houang@msu.edu>; reckase@msu.edu; knaebleb@uwstout.edu; dutters@uwstout.edu; mcavoy@npg.wustl.edu; sgupta@msu.edu; ricklaux@psu.edu; dlynch@bus.wisc.edu; JMathieu@business.uconn.edu; deshon@msu.edu; carms@wharton.upenn.edu; d.petmezas@surrey.ac.uk; 'liuyqphilly@gmail.com'; 'Sarah Galey' <sarahgaley@gmail.com>; 'Jihyun Kim' <toveave@gmail.com>; 'qinyun Lin' <linqinyun1001@gmail.com>; 'Aaron Zimmerman' <aaron.samuel.zimmerman@gmail.com>; 'Kaitlin Obenauf' <torphyka@gmail.com>; 'loyunjia@msu.edu'; 'I-Chien Chen' <ichiench@msu.edu>; 'Kim Jansen' <jansenki@msu.edu>; 'zixi chen' <zixichen68@gmail.com>; 'litenglo@msu.edu'; 'John Lane' <lanejoh3@msu.edu>; 'Guan K. Saw' <sawguan@msu.edu>; 'Soobin Jang' <soobinjang@gmail.com>; 'kmartin@msu.edu'; 'Ran Xu' <ranxu@msu.edu>; 'Christopher Lee' <cdeanlee@umich.edu>; 'Rachel White' <whitera3@msu.edu>; Angelo Joseph Garcia <angelo.giuseppe63@gmail.com>; 'Tingqiao Chen' <chentin4@msu.edu>; linamav@ime.usp.br; aansari@utexas.edu; pgoldsmith@tamu.edu; vpanov@ecko.uran.ru; astghik.mavisakalyan@curtin.edu.au; Youngs, Peter Alexander (pay2n <pay2n@eservices.virginia.edu>; Susanna Loeb <sloeb@stanford.edu>; sean.reardon@stanford.edu; Guanglei Hong <ghong@uchicago.edu>; sraudenb@uchicago.edu; kfinnigan@warner.rochester.edu; let2119@tc.columbia.edu; wendychan2016@u.northwestern.edu; yp2266@tc.columbia.edu; estuart@jhsph.edu; croninge@umd.edu; Christina Prell <cprell@umd.edu>; jqeaston@spencer.org; lauramd@gse.upenn.edu; jons@gse.upenn.edu; slu@ccsr.uchicago.edu; james_kim@gse.harvard.edu; jessaca.spybrook@wmich.edu; rmaynard@gse.upenn.edu; ssadoff@ucsd.edu; hmay@udel.edu; McCaffrey, Daniel F <dmccaffrey@ets.org>; marisa.a.cannata@vanderbilt.edu; gzamarro@uark.edu; psteiner@wisc.edu; bschneid@msu.edu; s.a.c.gorard@durham.ac.uk; amanda.carrico@vanderbilt.edu; omar.asensio@ucla.edu; machteld.vandecandelaere@ppw.kuleuven.be; William Penuel <William.Penuel@colorado.edu>; Allison Atteberry <Allison.Atteberry@colorado.edu>; Henry, Adam D - (adhenry <adhenry@email.arizona.edu>; Adam Gamoran <agamoran@Wtgrantfdn.org>; Eric Grodsky <egrodsky@ssc.wisc.edu>; 'Jim Spillane' <j-spillane@northwestern.edu>; s-ispa-landa@northwestern.edu; Rob Greenwald <wald@sree.org>; ittner@wharton.upenn.edu; michelsj@wharton.upenn.edu; Miller, Jason <mill2831@broad.msu.edu> 'Crosnoe, Robert' <crosnoe@austin.utexas.edu>; rwerum2@unl.edu; Muller, Chandra L <cmuller@austin.utexas.edu>; bollen@unc.edu; simon.cheng@uconn.edu; drew@drewdimmery.com; bradley.blaylock@okstate.edu; fabiog@ntu.edu.sg; shevlin@u.washington.edu; ben.b.hansen@umich.edu; chosman@umich.edu; paul.holland200@att.net; efjuncai@cityu.edu.hk; fbrykho@cityu.edu.hk; mjs3@princeton.edu; gabriela.olivares@uni.edu; klugman@temple.edu; mgaddis@live.unc.edu; Arthur.Kraft.1@city.ac.uk; HuaiZhang@ntu.edu.sg; jeeseonkim@wisc.edu; michael.weiss@mdrc.org; mgangl@ssc.wisc.edu; dharding@umich.edu; dlarcker@stanford.edu; trusticus@london.edu; tad61@columbia.edu; mfeng@katz.pitt.edu; chanli@katz.pitt.edu; gpainter@usc.edu; mcorreia@london.edu; jcore@mit.edu; ksmith1@unl.edu; ssbsup@okstate.edu; Spiro Maroulis <Spiro.Maroulis@asu.edu>; 'Ben Kelcey' <ben.kelcey@gmail.com>; 'Minh Duong' <mdthetahat@gmail.com>; mseltzer@ucla.edu; cwinship@wjh.harvard.edu; 'James Moody' <jmoody77@soc.duke.edu>; 'William Carbonaro' <William.Carbonaro.1@nd.edu>; 'Mark Berends' <Mark.Berends.3@nd.edu>; Min Sun <misun@uw.edu>; Richard.Taffler@wbs.ac.uk; machteld.vandecandelaere@ppw.kuleuven.be; annowens@usc.edu; keongkim@cityu.edu.hk; smithas@rhsmith.umd.edu; jwhitaker@richmond.edu; pkr@pkr.in; eminor1@nl.edu; Kaitlin Torphy <torphyka@gmail.com>; I-Chien Chen <ichiench@msu.edu>; loyunjia@msu.edu; liuyqphilly@gmail.com; Sarah Galey <sarahgaley@gmail.com>; Jihyun Kim <toveave@gmail.com>; qinyun Lin <linqinyun1001@gmail.com>; Aaron Zimmerman <aaron.samuel.zimmerman@gmail.com>; Kim Jansen <jansenki@msu.edu>; zixi chen <zixichen68@gmail.com>; litenglo@msu.edu; John Lane <lanejoh3@msu.edu>; Guan K. Saw <sawguan@msu.edu>; kmartin@msu.edu; Ran Xu <ranxu@msu.edu>; Christopher Lee <cdeanlee@umich.edu>; Rachel White <whitera3@msu.edu>; Angelo Joseph Garcia <angelo.giuseppe63@gmail.com>; Tingqiao Chen <chentin4@msu.edu>; Yincheng Ye <yycheng@vt.edu>; r-korajczyk@kellogg.northwestern.edu; R.H.Cuijpers@tue.nl; liuyqphilly@gmail.com; ranxu@msu.edu; sarahgaley@gmail.com; toveave@gmail.com; chentin4@msu.edu; linqinyun1001@gmail.com; aaron.samuel.zimmerman@gmail.com; torphyka@gmail.com; loyunjia@msu.edu; ichiench@msu.edu; croninge@umd.edu; 刘雨甲 <yujia.liu@bnu.edu.cn>; djklasik@umd.edu; Ruth N. Lopez Turley <turley@rice.edu>; David Williamson Shaffer <david.williamson.shaffer@gmail.com>; dkaplan@education.wisc.edu; Eric Camburn <ecamburn@education.wisc.edu>; Geoffrey Borman <gborman@education.wisc.edu>; georgek@uic.edu; stephen.morgan@jhu.edu; Cavanagh, Shannon E <scavanagh@austin.utexas.edu>; Dedrick, Robert <Dedrick@usf.edu>; John Lockwood <jrlockwood@ets.org>; Lou Mariano <Lou_Mariano@rand.org>; Joshua Cowen <jcowen@msu.edu>; pay2n@virginia.edu; T Gmail <tdietzvt@gmail.com>; Yu Xie <yuxie@umich.edu>; smithas@rhsmith.umd.edu; Konstantopoulos, Spyros <spyros@msu.edu>; Kimberly S. Maier <kmaier@msu.edu>; Schmidt, William <bschmidt@msu.edu>; Houang, Richard <houang@msu.edu>; reckase@msu.edu; knaebleb@uwstout.edu; dutters@uwstout.edu; mcavoy@npg.wustl.edu; sgupta@msu.edu; ricklaux@psu.edu; dlynch@bus.wisc.edu; JMathieu@business.uconn.edu; deshon@msu.edu; carms@wharton.upenn.edu; d.petmezas@surrey.ac.uk; 'liuyqphilly@gmail.com'; 'Sarah Galey' <sarahgaley@gmail.com>; 'Jihyun Kim' <toveave@gmail.com>; 'qinyun Lin' <linqinyun1001@gmail.com>; 'Aaron Zimmerman' <aaron.samuel.zimmerman@gmail.com>; 'Kaitlin Obenauf' <torphyka@gmail.com>; 'loyunjia@msu.edu'; 'I-Chien Chen' <ichiench@msu.edu>; 'Kim Jansen' <jansenki@msu.edu>; 'zixi chen' <zixichen68@gmail.com>; 'litenglo@msu.edu'; 'John Lane' <lanejoh3@msu.edu>; 'Guan K. Saw' <sawguan@msu.edu>; 'Soobin Jang' <soobinjang@gmail.com>; 'kmartin@msu.edu'; 'Ran Xu' <ranxu@msu.edu>; 'Christopher Lee' <cdeanlee@umich.edu>; 'Rachel White' <whitera3@msu.edu>; Angelo Joseph Garcia <angelo.giuseppe63@gmail.com>; 'Tingqiao Chen' <chentin4@msu.edu>; linamav@ime.usp.br; aansari@utexas.edu; pgoldsmith@tamu.edu; vpanov@ecko.uran.ru; astghik.mavisakalyan@curtin.edu.au; Youngs, Peter Alexander (pay2n <pay2n@eservices.virginia.edu>; Susanna Loeb <sloeb@stanford.edu>; sean.reardon@stanford.edu; Guanglei Hong <ghong@uchicago.edu>; sraudenb@uchicago.edu; kfinnigan@warner.rochester.edu; let2119@tc.columbia.edu; wendychan2016@u.northwestern.edu; yp2266@tc.columbia.edu; estuart@jhsph.edu; croninge@umd.edu; Christina Prell <cprell@umd.edu>; jqeaston@spencer.org; lauramd@gse.upenn.edu; jons@gse.upenn.edu; slu@ccsr.uchicago.edu; james_kim@gse.harvard.edu; jessaca.spybrook@wmich.edu; rmaynard@gse.upenn.edu; ssadoff@ucsd.edu; hmay@udel.edu; McCaffrey, Daniel F <dmccaffrey@ets.org>; marisa.a.cannata@vanderbilt.edu; gzamarro@uark.edu; psteiner@wisc.edu; bschneid@msu.edu; s.a.c.gorard@durham.ac.uk; amanda.carrico@vanderbilt.edu; omar.asensio@ucla.edu; machteld.vandecandelaere@ppw.kuleuven.be; William Penuel <William.Penuel@colorado.edu>; Allison Atteberry <Allison.Atteberry@colorado.edu>; Henry, Adam D - (adhenry <adhenry@email.arizona.edu>; Adam Gamoran <agamoran@Wtgrantfdn.org>; Eric Grodsky <egrodsky@ssc.wisc.edu>; 'Jim Spillane' <j-spillane@northwestern.edu>; s-ispa-landa@northwestern.edu; Rob Greenwald <wald@sree.org>; ittner@wharton.upenn.edu; michelsj@wharton.upenn.edu; Miller, Jason <mill2831@broad.msu.edu>