model averaging beyond model uncertainty in risk analysis n.
Skip this Video
Loading SlideShow in 5 Seconds..
Model Averaging: Beyond Model Uncertainty in Risk Analysis * PowerPoint Presentation
Download Presentation
Model Averaging: Beyond Model Uncertainty in Risk Analysis *

Loading in 2 Seconds...

play fullscreen
1 / 60

Model Averaging: Beyond Model Uncertainty in Risk Analysis * - PowerPoint PPT Presentation

  • Uploaded on

Model Averaging: Beyond Model Uncertainty in Risk Analysis *. Matthew W. Wheeler NIOSH *The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the National Institute for Occupational Safety and Health.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Model Averaging: Beyond Model Uncertainty in Risk Analysis *' - oriole

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
model averaging beyond model uncertainty in risk analysis

Model Averaging: Beyond Model Uncertainty in Risk Analysis *

Matthew W. Wheeler


*The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the National Institute for Occupational Safety and Health

  • A. John Bailer

Miami University/NIOSH

  • Introduction/Motivation.
  • Model Averaging 101.
  • Validation Simulation Study.
  • MA Software for dichotomous response.
introduction motivation
  • Model choice is frequently often a point of contention among risk assessors/risk managers.
  • Frequently multiple models describe the data “equivalently.”
  • Two model’s estimates of risk, especially at the lower bound, can differ dramatically.
  • Model uncertainty is inherent in most risk estimation, though practically ignored in most situations.
introduction motivation cont
Introduction/Motivation (cont)
  • Consider the problem of estimating a benchmark dose (BMD) from dichotomous dose response data.
  • Here we seek to estimate the BMD from a “plausible” model, given experimental data.
  • In these experiments:
    • Animals are exposed to some potential hazard.
    • The adverse response is assumed to be distributed binomially.
    • Risk (i.e, probability of adverse response) is estimated using regression modeling.
    • Multiple dose-response models can be used to estimate risk.

Common Dose-Response Models Used:

  • logistic model: (1)
  • log-logistic model: (2)
  • gamma: (3)
  • multistage (4)
  • probit (5)

Common Dose-Response Models Used:

  • log-probit (6)
  • quantal-linear (7)
  • quantal-quadratic (8)
  • Weibull (9)
  • where Γ(α)= gamma function evaluated at α, for Ф(x) = CDF N(0,1) and πi = γ when di=0 for models (2) and (7).
benchmark dose estimation
Benchmark dose estimation
  • BMD is the dose associated with the a specified increase in response relative to the control response (BMR); e.g.,:

dose d such that: BMR=[πd- π0]/[1- π0] or

BMR= πd- π0

  • The BMR is commonly set at values of 1%, 5%, 10%.
  • BMDL = 100(1-α)% lower confidence limit on the BMD.

NOTE: As πd is dependent on a model thus the BMD is model dependent!!

typical risk estimation process
Typical Risk Estimation Process
  • Given data (in absence of mechanistic information), a typical analyst will:
    • Estimate the regression coefficients for models (1)-(9).
    • Estimate the BMD/BMDL given the model.
    • Pick the “best model.”
example of this
Example of this:
  • Consider TiO2 lung tumor data which has been combined from the studies of Heinrich et al. (1995) Muhle et al. (1991) and Lee et al. (1985).
  • Here the benchmark dose (BMD) as well as its lower bound (BMDL) are estimated at BMRs of 10 and 1%.

** All fits were obtained using the US EPA’s BMDS

‡ Calculated using the number of parameters vs. the number of non-bounded parameters.

model choice
Model Choice
  • If we pick best AIC the quantal-quadratic model risk estimates would be chosen.
  • If we pick the best Pearson Χ2 test statistic the 3-degree multistage model would be chosen.
  • All of the models are reasonable based on these statistics.
model choice cont
Estimates are “reasonably similar” at the 10% estimate.

Estimates vary by a factor of 5 for the 1% estimate.

If a BMR of 0.1% is used (results not shown) the Ti02 BMD/BMDL estimates differ by a factor of 35.

This heterogeneity in model estimates exists even though the model fit statistics are very similar.

Model uncertainty (and heated debate) result when any one of the above models is chosen.

Model Choice (cont)
model averaging
Model Averaging
  • A better way would be to find an adequate way to combine all estimates, and thus describe/account for model uncertainty.
  • Model Averaging (MA) is one such method that may satisfactorily account for model uncertainty.
  • Instead of focusing on a single model it allows researchers to focus on “plausible behavior.”
model averaging cont
Model Averaging (cont.)
  • We can think of any model contributing information (including possible bias) to an analysis.
  • Picking any one model ignores other plausible information, and possibly introduces bias into the analysis.
  • Model averaging is a method that attempts to synthesize all of the information available.
model averaging cont1
Model Averaging (cont)
  • Kang et al. (Regulatory Toxicology and Pharmacology, 2000) and Bailer et al. (Risk Analysis,2005) proposed model averaging for risk assessment.
  • They used an “Average-BMD” methodology.
average dose ma
In this situation the benchmark dose is computed as :

Here the BMDk is estimated from one of K models individually

The BMDL is computed as:

Weights are based upon the formula:

Where Ii=AIC, Ii=KIC , or Ii=BIC. Other weights are possible.

“Average Dose” MA
average dose ma1
“Average Dose” MA
  • This method was found to perform unreliably in many situations in terms of nominal coverage and bias. (Wheeler and Bailer, Environmental and Ecological Statistics, In Press).
  • The results suggested a different BMD estimation technique may perform better. We call this the “average model” MA.
average model ma
“Average-Model” MA
  • Given the fits of models (1)-(9) MA:
    • Calculates the dose-response based upon a weighted average of dose-responses Raftery et al. (1997), Buckland et al. (1997)
    • Estimates the MA dose-response curve as:
    • Weights are formed as above.
average model ma benchmark dose
“Average-Model” MA Benchmark Dose
  • Given this “Average-model,” the benchmark dose is then computed by finding the dose that satisfies the equation

BMR = [πMA(d)i- πMA(0)]/[1- πMA(0)].

  • The BMDL is computed through a parametric bootstrap. Here the 5th percentile of the bootstrap distribution is used to compute the 95% lower tailed confidence limit estimate on the BMD.
average model vs average dose
“Average-Model” vs. “Average-Dose”
  • This is substantially different from the “average-dose” method.
    • “Average-Dose:”

Model Fits → Individual BMD estimation → BMD MA estimate

    • “Average-Model:”

Model Fits → MA Model Estimate → MA-BMD estimation

validation study
Validation Study
  • MA seems like a good idea, however we need to know if it works well in practice.
  • A simulation study was conducted to investigate the behavior of MA.
validation study1
Validation Study
  • A Monte Carlo simulation study was conducted to investigate the behavior of “average model” model averaging.
  • 54 true model conditions, using models (1) – (9) were used in the simulation.
  • Full study described in Wheeler and Bailer (Risk Analysis, in Press)

Step 1: Decide upon “TRUTH”

Truth was determined by using various response patterns. These were chosen to reflect shallow, medium, and steep curvature as well as low and high background response rates. In all six response patterns were chosen.


Step 1 (cont)

  • Using these 6 dose-response patterns the parameters associated with models (1)-(9) (e.g. γ,α,β,θ,θ2 ) were determined. For example:
  • These fits were used as the true underlying model in the simulation.
  • Three points allowed a unique specification of all models.
  • A variety of dose-response patterns were represented by the 54(6 conditions x 9 models) true dose response curves.

Step 2: Given truth set up simulation conditions.

  • The simulation proceeded by generating hypothetical toxicology experiments with response probability π(d).
  • With π(d) specified by one of the 54 true dose-response curves estimated in step 1.
  • These experiments consisted of 4 dose group design with doses of 0, 0.25, 0.50, and 1.0.
  • n=50 for all dose groups.
  • 2000 experiments were generated per true dose-response curve.
  • Bias as well as coverage [i.e., Pr(BMDL ≤ BMDtrue)] was estimated.
  • Coverage is reported here.

Step 3: Gather appropriate statistics.

  • In each experiment the “average-model” BMD as well as the BMDL was estimated.
  • BMRs of 1% and 10% were used to estimate the BMD.
  • Two model spaces for averaging were considered.
    • One space consisted of three flexible models: the multistage, Weibull and the log-probit model.
    • The second space had seven models that added the probit, logistic, quantal-linear, and quantal-quadratic to the three model space.
  • The simulation took approximately 1 CPU year of computation.
  • The BMDL was computed [with α = 0.05] and compared to the true benchmark dose at the specified BMR.
  • Coverage probability [i.e., Pr(BMDL ≤ BMDtrue)] was estimated across 2000 simulations.
coverage summary
Coverage (Summary)
  • Nominal coverage is reached for most simulation conditions.
  • MA fails to reach nominal coverage in the quantal-linear and similar cases.
quantal linear problems
Quantal-Linear Problems
  • The quantal-linear case represents a cause of concern if MA is to be used.
  • It is important to understand when (and why) it doesn’t work, and compare the method to current practice.
  • We first compare it to the common practice of picking the “best model.”
  • The “best-model” is chosen by a chi-squared goodness of fit statistic.

Coverage Comparison

MA vs. Best Model

*True model included in the model set.

ma vs best model
MA vs. “Best Model”
  • As a decision rule, in terms of coverage, “Average-model” 3-model averaging performs better than picking the best fitting model.

NOTE: The true model was included in the set of models the “best model” was chosen from.

further analysis of quantal linear model
Further Analysis of Quantal Linear Model
  • Despite the above it is disconcerting that a set of flexible models like the 3-model space model average fails to describe quantal-linear behavior.
  • We look at the sampling distribution of the experiment for the quantal-linear model, in attempt to explain this behavior.
further analysis of quantal linear model1
Further Analysis of Quantal Linear Model
  • The flexibility of the models combined with the sampling distribution introduces bias into the estimation of the dose-response curve.
  • The bias carries through in BMD estimation.
  • This also may be the cause of the conservative behavior (i.e. coverage > 99%) seen in the quantal-quadratic case.
bias corrected intervals
Bias Corrected Intervals
  • The evident bias suggests a search for a corrective mechanism.
  • Percentile based bootstrap confidence intervals are not optimal.
  • Biased Corrected and Accelerated (BCa) confidence intervals (CI) are preferred (Efron, 1994)
  • The BCa CIs are computationally more difficult. (i.e., full simulation becomes longer if these were being computed in all conditions)
  • Only the quantal-linear conditions were analyzed using the BCa confidence interval.
BCa CIs fail to be appropriate in shallow dose response cases.
  • In these cases an infinite BMD can be estimated through the bootstrap re-samples.
  • This causes a problem for the BCa technique, as it violates its underlying assumptions.
  • These problems are identifiable through histogram plots of the bootstrap resample.

Sampling distribution where BCa BMDLfails to cover the true BMD when the percentile based method does.

simulation conclusions
Simulation Conclusions
  • Model averaging recovers the central estimate, as well as the lower bound in most cases. (Some bias is however evident)
  • BMDs and BMDLs with BMRs of 1% are most problematic.
  • Problems exhibited with true quantal-linear dose-response mechanisms.
  • BCa confidence intervals can help remediate some of these problems.
“Average-model” model averaging is superior to picking the best model.
  • It recovers the true BMD uncertainty (in terms of correct coverage probabilities) when the true model is not in the MA model space.
  • The results show MA is not a panacea, it is however a step in the right direction.
model averaging software
Model Averaging Software
  • The results are promising but implementation of this approach is difficult.
  • The simulation code has been repackaged to allow users to implement dichotomous dose-response model averaging.
  • This is done in a simple MS Windows command prompt program.
Software is described that will implement Model Averaging for Dichotomous dose-response data (affectionately named MADr for short).
  • The software, as well as the code, is freely available under the GNU public license.
  • It will be available, in the near future, on NIOSH’s web site. A beta version is distributed with this talk.
madr bmd algorithm overview
MADr-BMD: Algorithm Overview


Data Input File

Model 1

Model K




Resampled Data

Model Average


Model 1

Model K




Bootstrap BMDL

Model Average



data specification
Data Specification
  • MADr is run on the command line by using the command “madr” at the prompt.
  • It’s input is a text file that can be edited using notepad or similar text editor.
  • By default it expects the file “input.txt”.
  • Alternate input file names can be specified.

For Example:

“madr tio2.txt”

“madr newtoxdata.txt”

program output
Program output
  • The output for the analysis is printed to the screen.
  • This can be redirected to a text file using the “>” command.

For Example:

“madr > output.txt”

“madr tio2.txt > anal1_output.txt”

Model Weight -2log(L) AIC BIC


Multistage 0.100 50.61 56.61 62.35

Logistic 0.228 50.96 54.96 58.79

Probit 0.218 51.05 55.05 58.88

Weibull 0.100 50.61 56.61 62.35

Log-Probit 0.122 50.21 56.21 61.95

Log-Logistic 0.121 50.23 56.23 61.97

Gamma 0.111 50.40 56.40 62.13



'Average-Model' Benchmark Dose Estimate


Nominally Specified Confidence Limit:0.950

Weighting Criterion: AIC

BMD Calculation: Added Risk

BMR: 0.100000

BMD: 0.894456594251



Acceleration: -0.034445

Bootstrap Resamples: 5000

Random Seed: 101


MODEL: Multistage, 2-degree polynomial:


Parameters Estimate StdErr


GAMMA: 0.171088 0.079273

BETA(1): 0.000000 N/A

BETA(2): 0.141611 0.052429

Optimization Succeeded


MODEL: Logistic:


Parameters Estimate StdErr


ALPHA: -1.824682 0.540209

BETA: 1.003971 0.301075

Optimization Succeeded


MODEL: Probit:


Parameters Estimate StdErr


ALPHA: -1.083244 0.300300

BETA: 0.590790 0.160163

Optimization Succeeded


  • Simulation Results + MADr implies dichotomous based model averaging can be used reliably in ones own research.
  • Though we have tested MADr, it is still a “use at your own risk program.”
As mentioned before model averaging is not a panacea.
  • As such it does not:
    • Relieve scientists from using their expert judgment.
    • Give automatic license to produce a low dose extrapolations.
    • Remove the need for adequate individual model fit diagnostics.
    • Remove all model uncertainty from the analysis.
It does:
    • Reframe the debate of model choice.
    • Produces relatively stable central estimates often independent of a given model being included in the average.
selected references
Selected References

Raftery, A. E. (1995). Bayesian model selection in social research. Sociological Methodology, 25,111-163.

Hoeting, J.A., Madigan, D., Raftery, A.E., & Volinsky, C.T. (1999) Bayesian model averaging: a tutorial. Statistical Science, 14, 382-417.

Buckland, S.T., Burnham, K. P., & Augustin, N. H., (1997). Model Selection: An Integral Part of Inference. Biometrics, 53, 603-618.

Kang, S.H., Kodell, R.L., Chen, J.J. (2000) Incorporating Model Uncertainties along with Data Uncertainties in Microbial Risk Assessment. Regulatory Toxicology and Pharmacology, 32, 68-72.

Bailer, A.J., Noble R.B. and Wheeler, M. (2005) Model uncertainty and risk

estimation for quantal responses. Risk Analysis,25,291-299.

Wheeler, M. W., & Bailer, A.J., (In Press). Properties of model-averaged BMDLs: A study of model averaging in dichotomous risk estimation. Risk Analysis