Kernel Stick-Breaking Process

1. Kernel Stick-Breaking Process D. B. Dunson and J. Park Discussion led by Qi An Jan 19th, 2007

2. Outline • Motivation • Model formulation and properties • Prediction rules • Posterior Computation • Examples • Conclusions

3. Motivation • Consider a problem of estimating the conditional density of a response variable using a mixture model, , where Gx is an unknown probability measure indexed by x. • The problem of defining priors for random probability measures on Gx has received increasing attention in recent year. For example, DP, DDP.

4. One model • In DDP, the atoms can vary with x according to a stochastic process while the weights are fixed • Dunson et al propose a model to allow the weights to vary with predictors while this model lacks reasonable marginalization and updating properties.

5. Model formulation • Introduce a countable sequence of mutually independent random components • The kernel stick-breaking process (KSBP) can be defined as follows:

6. About the model • The model for Gx is a predictor-dependent mixture over an infinite sequence of basis probability measures, Gh* located at Γh. • Bases located close to x and having a smaller index, h, tend to receive higher probability weight. • KSBP accommodates dependency between Gx and Gx’

7. Special cases • If K(x,Γ)=1 for all and Gh*~DP(αG0), it is a stick-breaking mixture of DP. • If K(x,Γ)=1, and , we obtain Gx≡G, with G having a stick-breaking prior. • If and , we obtain a Pitman-Yor process.

8. Properties • Let , we can obtain • The correlation between measures First moment No dependency on V and Γ Second moment It can be proven and the value 1 in the limit as x x’ where

9. Alternative representation The KSBP has an alternative representation The moments and correlation coefficient has the form

10. Truncation • For stick-breaking Gibbs sampler, we need to make truncation approximation • Author proves that the residual weights decrease exponentially fast in N and an accurate approximation may be obtained for moderate N The approximated model can be expressed as

11. Prediction rules • Consider a special case in which The model can be equivalently expressed as:

12. Prediction rules • Define and is a subset of the integers between 1 and n • It can be proven that the probability that subjects i and j belong to the same cluster is The predictive distribution is obtained by marginalization where and denote the set of possible r- dimensional subsets of {1,…,s} that include i

13. Posterior Computation From the prior, we can obtain 1, sample Si 2, sample CSi when Si=0 (assign subject I to a new atom at an occupied location) 3, sample θh

14. 4, sample Vh 5, sample Γh using a Metropolis-Hastings step or Gibbs step if H is a set of discrete potential locations First sample and then, alternate between (i) Sampling (Aih,Bih) from their conditional distribution (ii) Updating Vh by sampling from conditional posterior

15. Simulated examples

18. Conclusions • This stick-breaking process is useful in setting in which there is uncertainty in an uncountable collection of probability measures • The process can be applied in predictor dependent clustering, dynamic modeling and spatial data analysis, besides the density regression. • The KSBP formulation can be applied to many tools developed for exchangeable stick-breaking processes with minimal modification. • A predicator dependent urn scheme is obtained, which generalizes the Polya urn scheme