Presented by: Mingyuan Zhou Duke University January 20, 2012

1. Characterizing the Function Space for Bayesian Kernel ModelsNatesh S. Pillai, Qiang Wu, Feng LiangSayan Mukherjee and Robert L. WolpertJMLR 2007 Presented by: Mingyuan Zhou Duke University January 20, 2012

2. Outline • Reproducing kernel Hilbert space (RKHS) • Bayesian kernel model • Gaussian processes • Levy processes • Gamma process • Dirichlet process • Stable process • Computational and modeling considerations • Posterior inference • Discussion

3. RKHS In functional analysis (a branch of mathematics), a reproducing kernel Hilbert space is a Hilbert space of functions in which pointwise evaluation is a continuous linear functional. Equivalently, they are spaces that can be defined by reproducing kernels. http://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space

4. A finite kernel based solution The direct adoption of the finite representation is not a fully Bayesian model since it depends on the (arbitrary) training data sample size . In addition, this prior distribution is supported on a finite-dimensional subspace of the RKHS. Our coherent fully Bayesian approach requires the specification of a prior distribution over the entire space H.

5. Mercer kernel

6. Bayesian kernel model

8. Properties of the RKHS

9. Properties of the RKHS

10. Bayesian kernel models and integral operators

12. Two concrete examples

13. Two concrete examples

14. Bayesian kernel models

15. Gaussian processes

16. Levy processes

17. Levy processes

18. Poisson random fields

19. Poisson random fields

20. Dirichlet Process

21. Symmetric alpha-stable processes

22. Symmetric alpha-stable processes

23. Computational and modeling considerations • Finite approximation for Gaussian processes • Discretization for pure jump processes

24. Posterior inference • Levy process model • Transition probability proposal • The MCMC algorithm

26. Classification of gene expression data

27. Classification of gene expression data

28. Discussion • This paper formulates a coherent Bayesian perspective for regression using a RHKS model. • The paper stated an equivalence under certain conditions of the function class G and the RKHS induced by the kernel. This implies: • (a) a theoretical foundation for the use of Gaussian processes, Dirichlet processes, and other jump processes for non-parametric Bayesian kernel models. • (b) an equivalence between regularization approaches and the Bayesian kernel approach. • (c) an illustration of why placing a prior on the distribution is natural approach in Bayesian non-parametric modelling. • A better understanding of this interface may lead to a better understanding of the following research problems: • Posterior consistency • Priors on function spaces • Comparison of process priors for modeling • Numerical stability and robust estimation