Data Mining and the OptIPuter

1. Data Mining and the OptIPuter Padhraic Smyth University of California, Irvine

2. Data Mining of Spatio-Temporal Scientific Data • Modern scientific data analysis • increasingly data-driven • data often consist of massive spatio-temporal streams • Research focus • characterizing spatio-temporal structure in data • statistical models for object shapes, trajectories, patterns... • data mining from scientific data streams (NSF, Optiputer) • recognition of waveforms in time-series archives (JPL,NASA) • inference of dynamic gene-regulation networks from data (NIH) • Markov models for spatio-temporal weather patterns (DOE) • clustering and modeling of storm trajectories (LLNL)

3. Image-voxel Data (“slices” of olfactory bulb in rats) Automatic segmentation of cellular structures of interest (glomelular layer) • Thematic maps • Data mining • Scientific discovery

4. Image-voxel Data (Remote sensing AVIRIS spectral data) Focus of attention on wavelengths of interest • Thematic maps • Data mining • Scientific discovery

5. What’s wrong with this information flow? • “One-way” • Flow of information is from data to scientist • Real scientific investigation is “two-way” • Scientist interacts, explores, queries the data • Most current data mining/analysis tools are relatively poor at handling interaction • Algorithms are “black-box”, do not allow scientists to be “in the loop” • Algorithms have no representation of the scientist’s prior knowledge or goals (no user models) • OptIPuter project • “next generation” data mining tools for effective exploration of massive 2d/3d data sets

6. OptIPuter focus in Data Mining • Data • 2d (or multi-d) spatio-temporal image/voxel data • Goals • Allow scientists to explore these massive data sets in an efficient and flexible manner leveraging the OptIPuter architecture • Produce interactive software tools that allow scientists to explore massive data in an interactive manner: • automated segmentation, thematic maps, focus of interest • Technical Challenges • Scaling statistical algorithms to massive data streams • Providing mechanisms for effective scientific interaction • Developing algorithms for automated “focus-of-attention”

7. Analysis of Extra-Tropical Cyclones [with Scott Gaffney (UCI), Andy Robertson (IRI/Columbia), Michael Ghil (UCLA)] • Extra-tropical cyclone = mid-latitude storm • Practical Importance • Highly damaging weather over Europe • Important water-source in United States • Scientific Importance • Influence of climate on cyclone frequency, strength, etc. • Impact of cyclones on local weather patterns

8. Sea-Level Pressure Data • Mean sea-level pressure (SLP) on a 2.5° by 2.5° grid • Four times a day, every 6 hours, over 20 years Blue indicates low pressure

9. Winter Cyclone Trajectories

10. Clustering Methodology • Mixtures of curves • model as mixtures of noisy linear/quadratic curves • note: true paths are not linear • use the model as a first-order approximation for clustering • Advantages • allows for variable-length trajectories • allows coupling of other “features” (e.g., intensity) • provides a quantitative (e.g., predictive) model • [contrast with k-means for example]

11. Clusters of Trajectories

12. Applications • Visualization and Exploration • improved understanding of cyclone dynamics • Change Detection • can quantitatively compare cyclone statistics over different era’s or from different models • Linking cyclones with climate and weather • correlation of clusters with NAO index • correlation with windspeeds in Northern Europe