Variable selection for tailoring treatment
1 / 36

Variable Selection for Tailoring Treatment - PowerPoint PPT Presentation

  • Uploaded on

Variable Selection for Tailoring Treatment. L. Gunter, J. Zhu & S.A. Murphy ASA, Nov 11, 2008. Outline. Motivation Need for Variable Selection Characteristics of a Tailoring Variable A New Technique for Finding Tailoring Variables Comparisons Discussion. Motivating Example.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Variable Selection for Tailoring Treatment' - donovan-williams

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Variable selection for tailoring treatment

Variable Selection for Tailoring Treatment

L. Gunter, J. Zhu & S.A. Murphy

ASA, Nov 11, 2008


  • Motivation

  • Need for Variable Selection

  • Characteristics of a Tailoring Variable

  • A New Technique for Finding Tailoring Variables

  • Comparisons

  • Discussion

Simple example
Simple Example

Nefazodone - CBASP Trial



Nefazodone + Cognitive

Behavioral Analysis

System of Psychotherapy


50+ baseline covariates, both categorical and continuous

Simple example1
Simple Example

Nefazodone - CBASP Trial

Which variables in X are important for tailoring the treatment?


  • We want to select the treatment that “optimizes” R

  • The optimal choice of treatment may depend on X


  • The optimal treatment(s) is given by

  • The value of d is

Need for variable selection
Need for Variable Selection

  • In clinical trials many pretreatment variables are collected to improve understanding and inform future treatment

  • Yet in clinical practice, only the most informative variables for tailoring treatment can be collected.

  • A combination of theory, clinical experience and statistical variable selection methods can be used to determine which variables are important.

Current statistical variable selection methods
Current Statistical Variable Selection Methods

  • Current statistical variable selection methods focus on finding good predictors of the response

  • Also need variables to help determine which treatment is best for which types of patients, e.g. tailoring variables

  • Experts typically have knowledge on which variables are good predictors, but intuition about tailoring variables is often lacking

What is a tailoring variable
What is a Tailoring Variable?

  • Tailoring variables help us determine which treatment is best

  • Tailoring variables qualitatively interact with the treatment; different values of the tailoring variable result in different best treatments.

    No Interaction Non-qualitative Interaction Qualitative interaction

Qualitative interactions
Qualitative Interactions

  • Qualitative interactions have been discussed by many within stat literature (e.g. Byar & Corle,1977; Peto, 1982; Shuster & Van Eys, 1983; Gail & Simon, 1985; Yusuf et al., 1991; Senn, 2001; Lagakos, 2001)

  • Many express skepticism concerning validity of qualitative interactions when found in studies

  • Our approach for finding qualitative interactions should be robust to finding spurious results

Qualitative interactions1
Qualitative Interactions

  • We focus on two important factors

    • The magnitude of the interaction between the variable and the treatment indicator

    • The proportionof patients for whom the best choice of treatment changes given knowledge of the variable

      big interaction small interaction big interaction

      big proportion big proportion small proportion

Ranking score s
Ranking Score S

  • Ranking Score:


  • S estimates the quantity described by Parmigiani (2002) as the value of information.

Ranking score s1
Ranking Score S

  • Higher Sscorescorrespond to higher evidence of a qualitative interaction between X and A

  • We use this ranking in a variable selection algorithm to select important tailoring variables.

    • Avoid over-fitting in due to large number of X variables

    • Consider variables jointly

Variable selection algorithm
Variable Selection Algorithm

  • Select important predictors of R from (X, X*A) using Lasso

    -- Select tuning parameter using BIC

  • Select all X*A variables with nonzero S.

    -- Use predictors from 1. to form linear regression estimator of to form S.

(using linear models)


  • Lasso on (X, A, XA) (Tibshirani, 1996)

    • Lasso minimization criterion:

      where Zi is the vector of predictors for patient i, λ is a penalty parameter

    • Coefficient for A not penalized

    • Value of λ chosen by Bayesian Information Criterion (BIC) (Zou, Hastie & Tibshirani, 2007)

Variable selection algorithm1
Variable Selection Algorithm

  • Rank order (X, X*A)variables selected in steps 1 & 2 using a weighted Lasso

    -- Weight is 1 if variable is not an interaction

    -- Otherwise weight for kth interaction is

    -- is a small positive number.

    -- Produces a combined ranking of the selected (X, X*A)variables (say p variables).

Variable selection algorithm2
Variable Selection Algorithm

  • Choose between variable subsets using a criterion that trades off maximal value of information and complexity.

    -- The ordering of the p variables creates p subsets of variables. Estimate the value of information for each of the p subsets

    -- Select the subset, k with largest


  • Data simulated under wide variety of realistic decision making scenarios (with and without qualitative interactions)

    • Used X from the CBASP study, generated new Aand R

  • Compared:

    • New method: S with variable selection algorithm

    • Standard method: BIC Lasso on (X, A, XA)

  • 1000 simulated data sets: recorded percentage of time each variable’s interaction with treatment was selected for each method

Simulation results
Simulation Results

* Over the total possible increase; 1000 data sets each of size 440

Simulation results1
Simulation Results

  • Pros: when the model contained qualitative interactions, the new method gave significant increases in expected response over BIC-Lasso

  • Cons: the new method resulted in a slight increase in the number of spurious interactions over BIC-Lasso

Nefazodone cbasp trial
Nefazodone - CBASP Trial

Aim of the Nefazodone CBASP trial – to compare efficacy of three alternate treatments for major depressive disorder (MDD):

  • Nefazodone,

  • Cognitive behavioral-analysis system of psychotherapy (CBASP)

  • Nefazodone + CBASP

    Which variables might help tailor the depression treatment to each patient?

Nefazodone cbasp trial1
Nefazodone - CBASP Trial

  • For our analysis we used data from 440 patients with

Method application and confidence measures
Method Application and Confidence Measures

  • When applying new method to real data it is desirable to have a measure of reliability and to control family-wise error rate

  • We used bootstrap sampling to assess reliability

    • On each of 1000 bootstrap samples:

      • Run variable selection method

      • Record the interaction variables selected

    • Calculate selection percentages over bootstrap samples

Error rate thresholds
Error Rate Thresholds

  • To help control family-wise error rate, compute the following inclusion thresholdsfor selection percentages:

    • Repeat 100 times

      • Permute interactions to remove effects from the data

        • Run method on 1000 bootstrap samples of permuted data

        • Calculate selection percentages over bootstrap samples

      • Record largest selection percentage over the p interactions

    • Threshold: (1-α)th percentile over 100 max selection percentages

  • Select all interactions with selection percentage greater than threshold

Error rate thresholds1
Error Rate Thresholds

  • When tested in simulations using new method, error rate threshold effectively controlled family-wise error rate

  • This augmentation of bootstrap sampling and thresholding was also tested on BIC Lasso and effectively controlled family-wise error rate in simulations

Nefazodone cbasp trial2
Nefazodone - CBASP Trial






  • This method provides a list of potential tailoring variables while reducing the number of false leads.

  • Replication is required to confirm the usefulness of a tailoring variable.

  • Our long term goal is to generalize this method so that it can be used with data from Sequential, Multiple Assignment, Randomized Trials as illustrated by STAR*D.

  • Email Susan Murphy at [email protected] for more information!

  • This seminar can be found at


  • Support: NIDA P50 DA10075, NIMH R01 MH080015 and NSF DMS 0505432

  • Thanks for technical and data support go to

    • A. John Rush, MD, Betty Jo Hay Chair in Mental Health at the University of Texas Southwestern Medical Center, Dallas

    • Martin Keller and the investigators who conducted the trial `A Comparison of Nefazodone, the Cognitive Behavioral-analysis System of Psychotherapy, and Their Combination for Treatment of Chronic Depression’

Lasso weighting scheme
Lasso Weighting Scheme

  • Lasso minimization criterion equivalent to:

    so smaller wj means greater importance

  • Weights where

    • vj = 1for predictive variables

    • vj = for prescriptive variables

Agv criterion
AGV Criterion

  • For a subset of k variables, X{k} the Average Gain in Value ( AGV) criterion is


  • The criterion selects the subset of variables with the maximum proportion of increase in E[R] per variable

Simulation results s score
Simulation Results (S-score)

×Qualitative Interaction

Spurious Interaction

×Qualitative Interaction

Non-qualitative Interaction

Spurious Interaction