Transforming the efficiency of Partial EVSI computation

1. Transforming the efficiency of Partial EVSI computation Alan Brennan Health Economics and Decision Science (HEDS) Samer Kharroubi Centre for Bayesian Statistics in Health Economics (CHEBS) University of Sheffield, England a.brennan@sheffield.ac.uk s.a.kharroubi@sheffield.ac.uk

2. Expected Value of Sample Information (EVSI) • EVSI works out the expected impact on decision making if we collect more data • We • Simulate a collected sample dataset • Update uncertainty in parameters given data • ? Choose a different decision option given data • Quantify increase in benefit over baseline decision • Repeat for many sample datasets • Calculate the expected increase in benefit

3. EVSI The Computational Problem • EVSI works out the expected impact on decision making if we collect more data • Conventional Computations required • “Outer” Monte Carlo sample • Bayesian Update – analytic or MCMC • “Inner” Monte Carlo sample e.g. 10,000 times • Evaluate each net benefit function each time • Repeat for many sample datasets e.g. 10,000 times • Total e.g. 100,000,000 evaluations of net benefit

4. Mathematical Notation EVSI = Expected Payoff given only current information Expectation over sampled datasets Expected Payoff for each Decision given particular new data Xi  = uncertain model parameters t = set of possible treatments (decision options) NB(d, ) = net benefit (λ*QALY – Cost) for decision d,  i = parameters of interest – possible data collection Xi = data collected on the parameters of interest i

5. Laplace approximation • Sweeting and Kharroubi (2003) developed a 2nd order approximation to evaluate the posterior expectation of any real valued smooth function v() with a vector of d uncertain parameters  given new available data X. ------ ---------------------------------- 1st order 2nd order term term

6. Eureka • For EVSI the first term in the formula is • We can adapt Laplace approximation to evaluate the EVSI innerexpectation ! -------- ----------------------------------------- 1st order 2nd order • Only requires 1+3d evaluations of net benefit (Kharroubi and Brennan 2005)

7. Univariate Explanation: + • + and - are 1 standard deviation away from the posterior mode  ^ ^ 

8. Univariate Explanation: α+ • α+ and α- are weights, functions of the ratio of the slopes of the log density function at θ+, θ- If distribution is symmetric then α+ = α- =½

9. Multivariate Requires Matrix Algebra for each dataset Xi • θi+, θi- are vectors. • each is the i th row of a matrix θ+, θ- • The first i -1 components are posterior modes θ1 ...θi-1 • i th is θi± (ki)-1/2 , where ki is 1/first entry of {J(i)}-1 • Remaining i +1 to d components are chosen to maximise the posterior density given the first i components • αi+ and αi- are vectors of weights, which are calculated based on partial derivatives of the log posterior density function at θi+, θi- • Requires numerical optimisation ^ ^ ^

10. Case Studies • Case Study 1 • 2 treatments – T1 versus T0 • Uncertainty in …… 19 independent parameters • Univariate Normal prior and data • Net benefit function is sum-product form • NB1=  (θ5θ6θ7+θ8θ9θ10) – (θ1+θ2θ3θ4 ) • Case Study 2 • Uncertainty in …… 19 correlated parameters • Multivariate Normal prior and data

11. Illustrative Model

12. Case Study 1 Results (5 sets) 1st order Laplace is accurate

13. Case Study 2: 1st order wrong 2nd order is accurate

14. Accuracy of inner integral approximation • Parameters 6,15 • Sample size n=50 • Out of 1000 datasets the resulting decision between 2 treatments was different in 7 • i.e. 0.7% error Laplace Monte Carlo

15. Trade-off in Computation Time

16. Computation Time What-If Analyses • Efficiency gain due to Laplace approximation increases rapidly as model run time for one evaluation of net benefit increases

17. Limitations • Any Type of Net benefit function • analytic function of model parameters • result of probabilistic model e.g. individual level simulation • Characterisation of Uncertainty • Need functional form for probability density function • Smooth and differentiable, • i.e. not just a histogram to sample from • write down the equations for posterior density function and its derivative mathematically

18. Conclusions • EVSI calculations using the Laplace approximation are in line with those using 2 level Monte-Carlo method in case studies so far • Method is very generalisable once you understand the mathematics and algorithm • Computation time reductions depend on times to compute net benefit functions

19. Thankyou • 'Wisest are they who know they do not know‘ • ‘Especially if they can calculate whether it’s worth finding out’

20. References • Brennan, A. B., Chilcott, J. B., Kharroubi, S. A, O'Hagan, A. A Two Level Monte Carlo Approach to Calculation Expected Value of Sample Information: How To Value a Research Design. Presented at the 24th Annual Meeting of SMDM, October 23rd, 2002, Washington. 2002. http://www.shef.ac.uk/content/1/c6/03/85/60/EVSI.ppt • Ades AE, Lu G, Claxton K. Expected value of sample information calculations in medical decision modelling. Medical Decision Making. 2004 Mar-Apr;24(2):207-27. • Sweeting, T. J. and Kharroubi, S. A.(2003). Some new formulae for posterior expectations and Bartlett corrections. Test,12(2): 497-521. • Kharroubi, S. A. and Brennan, A. (2005). A Novel Formulation for Approximate Bayesian Computation Based on Signed Roots of Log-Density Ratios. Research Report No. 553/05, Department of Probability and Statistics, University of Sheffield. Submitted to Applied Statistics. http://www.shef.ac.uk/content/1/c6/02/56/37/Laplace.pdf