1 / 47

Clustering Functional Data: Methods and Applications

Clustering Functional Data: Methods and Applications. Catherine Sugar University of Southern California sugar@usc.edu This is joint work with Gareth James of USC UCLA May 1st, 2006. Clustering and Functional Data.

lee
Download Presentation

Clustering Functional Data: Methods and Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Clustering Functional Data:Methods and Applications Catherine Sugar University of Southern California sugar@usc.edu This is joint work with Gareth James of USC UCLA May 1st, 2006

  2. Clustering and Functional Data • Cluster Analysis: The art of finding groups in data. Points in the same cluster should be as similar as possible and points in disjoint clusters should be widely separated • Functional Data: Observations for a subject consist of curves or trajectories rather than finite dimensional vectors. • Growth curves • Longitudinal measurements of clinical status • Technology evolution • Spectra

  3. Outline • Traditional approaches to clustering curves and problems with sparse data • A new approach using basis functions and a mixture model • Applications of our approach in medicine and business • Tools, extensions, and model selection issues

  4. Functional Examples Spinal Bone Mineral Density Data Technology Evolution Curves

  5. Functional Examples: Membranous Nephropathy Data

  6. Traditional Approaches To Functional Clustering • Regularization: • Form a grid of equally spaced time points. • Evaluate each curve at the time points, giving a finite representation of each curve. • Apply a standard finite dimensional method possibly with a regularization constraint • Filtering: • Fit a smooth curve to each subject using a finite set of basis functions, • Perform clustering on the basis coefficients ()

  7. Problems With the Traditional Approaches • Regularization: • Cannot be easily applied when curves are measured at different or unevenly spaced time points or when the data are too sparse • Even when it can be used, the resulting data vectors are high-dimensional and auto-correlated • Filtering: • Measurements may be too sparse to fit a curve for each subject • Requires fitting many parameters • If subjects are measured at different time points, the basis coefficients will not have a common covariance

  8. Our Model • Let gi(t),Yi(t) and i(t) respectively be the true value, observed value and error for ith curve at time t. i.e. • We represent g(t) using a natural cubic spline basis: where s(t) is a spline basis vector and i is the vector of spline coefficients. • The coefficients are treated as random effects with where zi denotes cluster membership

  9. Our Model • Our model becomes • We fit this model using the observed time points and an EM algorithm.

  10. Fitting The Model:Bone Density Data In Two Clusters

  11. Fitting The Model: Technology Data In Two Clusters

  12. Model Applications I:Low Dimensional Representations • One can plot functional data but it is hard to assess relative “distances” between curves • We use the basis coefficients to project data into a low-dimensional space where it can be plotted as points • Projecting causes no information loss in terms of cluster assignment • The projections are exact analogues of the discriminants used in LDA

  13. Model Applications I:Low Dimensional Representations

  14. Model Applications ILow Dimensional Representations: Bone Data

  15. Model Applications I:Low Dimensional Representations: Technology Data

  16. Model Applications I:Low Dimensional Representations: Nephrology Data

  17. Model Applications II:Dimensions of Separation • It is useful to know what dimensions do the best job of separating the clusters. • This depends on a combination of distance between clusters and within cluster covariance, and is equivalent to identifying which dimensions determine cluster assignment • The optimal weights for cluster assignment are given by an extension of the classical discriminant function for a Gaussian mixture:

  18. Model Applications II:Dimensions of Separation Correlation and Covariance: Discriminating Functions:

  19. Model Applications III:Prediction and Confidence Intervals • Another advantage of our method is that it provides accurate predictions for missing portions of g(t) • Natural estimate: • The prediction with minimum mean squared error is • CI’s and PI’s: Two step procedure—find the set of clusters most likely to contain g(t) and then create intervals conditional on cluster membership

  20. Model Applications IIIPrediction on Bone Data

  21. Model Applications IIIPrediction on Technology Data Optical bit density Magnetic storage bit density

  22. HDD 3.5 in. storage capacity • Black = Functional Clustering • Red = Linear Gompertz • Green = Mansfield-Blackman • Cyan = Weibull • Orange = Bass • Blue = S-curve

  23. A Comparison With Standard Approaches • We took the first 10 years as training data and tried to predict the following 5 years using various different approaches. • Here we report the MSE on the left out data as a percentage of that from using a traditional S-curve (logistic curve).

  24. Advantages of Our Model • Borrows strength from all curve fragments to simultaneously estimate mixture parameters and requires fitting fewer parameters. • Allows one to make more accurate predictions into the future based on only a few observations. • Flexible. Can be used effectively when data are sparse, irregularly spaced or sampled at different time points • Automatically puts the correct weights on estimated basis coefficients • Can be easily extended to include multiple functional and finite dimensional covariates.

  25. Extensions I:Multiple Functional Covariates • Just as finite dimensional clustering algorithms can incorporate multiple covariates one should be able to use multiple functional variables • We can do this creating a block diagonal spline basis matrix using the entries for the p individual curves: • More care must be taken with the error structure but the same basic model and fitting procedure apply.

  26. Extensions II:Finite Dimensional Covariates • It is just as easy to add finite dimensional covariates to the model • Let Xi be the vector of finite dimensional covariates. • We replace the spline basis matrix, Si, by the identity, Iix • The model can be fit just as before • Note that this provides a way of doing high dimensional standard clustering problems with missing data—just delete the corresponding rows of the identity matrix

  27. Extensions III:Dimension Reduction • Reducing dimensions ahead of time (e.g. by PCA) may be risky. • Example below shows a case where the dimensions that explain most of the variability are not the ones determining cluster separation. Our method (right) does a superior job

  28. References: • Bacrach, L. et al.(1999) Bone mineral Acquisition in healthy Asian, Hispanic, Black, and Caucasian youth; a longitudinal study. Journal of Clinical Endocrinology & Metabolism 84, 4702-4712 • Banfield, J. and Raftery, A. (1993). Model-based Gaussian and non-gaussian clustering. Biometrics 49, 803-821 • James, G., and Hastie, T. (2001). Functional Linear Discriminant analysis for irregularly sampled curves. JRSSB 63, 533-550 • James, G., Hastie, T., and Sugar, C. (2000). Principal component models for sparsely sampled functional data. Biometrika87, 587-602 • James, G. and Sugar, C. (2003) Clustering for sparsely sampled functional data. JASA, 98, 397-408 • Sugar, C., and James, G. (2003) Finding the number of clusters in a data set: An information theoretic approach. JASA, 98, 750-763

  29. Model Selection Issues: • Choosing the spline basis and number and placement of knots • Choosing the dimension of the mean space • Choosing the covariance structure for the clusters • Choosing the number of clusters

  30. How Many Clusters: • Raftery et al. suggest using approximate Bayes factors in the finite dimensional setting • We propose an approach based theory from Electrical Engineering involving distortion • Distortion is • Plot distortion as a function of k, the number of clusters • Rate distortion theory suggests the form of the resulting “distortion curve”

  31. How Many Clusters:Basic Results • If the data are generated from a single cluster in q dimensions then asymptotically the distortion curve can be made linear, specifically • When there are an unknown number, K, of clusters, the inverse distortion plot will be straight both before and after K, and will experience its maximum jump at K subject to certain conditions.

  32. How Many Clusters:Examples The figures below show a transformed distortion curve when there is (a) a single component and (b) six components in the generating mixture distribution

  33. A General Functional Model • Let g(t) be the curve for a randomly chosen individual. (We will assume g(t) follows a Gaussian process.) • If g(t) is in the kth cluster we write • If Y is the vector of observed values at times t1,…,tn and errors are assumed independent then

  34. A General Functional Model • Regularization and filtering can be viewed as approaches to fitting the general functional clustering model. • The regularization approach estimates k(t) and k(t,t') on a fine grid of time points with structural constraints on k(t,t') • The filtering approach assumes g(t) = (t) where (t) is a vector of basis functions and  is the vector of coefficients. The 's are estimated separately for each individual and then clustered

  35. Our Model • It is further useful to parameterize the cluster mean coefficients as where 0 and k are respectively q and h dimensional vectors and  is a qh matrix. • If h < K-1 then the we are assuming the cluster mean coefficients lie in a restricted subspace. • Our model becomes

  36. Fitting Our Model Via EM • Fitting our model involves estimating • We can do this by maximizing either the classification likelihood or the clustering likelihood, noting that conditional on class membership • We can use an iterative procedure such as EM to obtain the estimates

  37. Model Selection III:Alternate Covariance Structures • So far we have assumed a common covariance matrix, , for all clusters • Raftery et al. suggest a class of covariance structures in their model-based clustering methods for finite-dimensional data: • James and Hastie suggest regularization or rank reduction in their papers on functional PCA and LDA

  38. High pressure sodium lamp

  39. HCF storage capacity

  40. Fluorescent lamp

  41. Inkjet printer resolution

  42. Fiber optics data transfer speed

  43. Wireless data transfer speed

  44. Not So Good: Incandescent lamp

  45. Metal halide lamps

  46. Zip drive storage capacity

  47. Cutting Edge Results

More Related