Data Exploration with DAVIS

1. Moon HUH1, KwangRyeol SONG2, YoungSuk PARK1, KyungWook Shim 1Sungkyunkwan University, Seoul, Korea 2 Kwansei Research Institute, Seoul, Korea Data Exploration with DAVIS Variable Selection

2. Purpose of DAVIS to visually explore the structure or pattern of data Variable Selection

3. Components of DAVIS Data Manipulation Statistical Tools Plots Graphic Controllers Variable Selection

4. Data Manipulation • Observation/variable selection • Focusing/deleting a subset of data set • Missing value process • Discretization Variable Selection

5. Plots - Univariate • Bar Charts • Histogram • QQ Plot • FEDF • BoxPlot • Parallel Coordinates Variable Selection

6. BoxPlot: Features • Standardization • Indentification Variable Selection

7. Parallel Coordinates: Features • Direction of Plotting: Horizontal / Vertical • Ordering of the Variables: Component / Permutation • Jittering Variable Selection

8. Parallel Coordinates -options Variable Selection

9. Plots-multivariate • Scatterplot • Loess curve fitting • Touring • Dendrogram • Line Mosaic Plot • PCA plot Variable Selection

10. Scatterplot-options Variable Selection

11. Touring –GrandTour/Tracking Variable Selection

12. Dendrogram –Agglomeration /Distance options Variable Selection

13. Line Mosaic Plot –for discrete data Variable Selection

14. PCA plot Variable Selection

15. Real time grouping with DAVIS - hiliting • Manually grouping the data set into 2 subsets by mouse brushing a subset of data • Always can go back to the original data set Variable Selection

16. Real time grouping with DAVIS–deleting/focusing Variable Selection

17. Interactive Clustering with DAVIS-linking Variable Selection

18. Clustering with DAVIS: EM with 3 groups Variable Selection

19. Coloring a subset –outlier detection Variable Selection

20. Touring with DAVIS- Tracking • Can investigate multidimensional structure of the data Variable Selection

21. Data exploration with Decision Trees-Titanic data Variable Selection

22. Decision Trees-2 Variable Selection

23. Variable selection with DAVIS • Target (Class) variable discrete (nominal) type • Candidate variables nominal, numerical, and complex type Variable Selection

24. Variable subset selection methods • MDI( Lee and Huh, 2003) . using p-values for the test statistics between the 2 variables. –log (p-value) is suggested • ReliefF (Kira and Randell, 1992) Relief (x)=P[different value of X | different class] -          P[different value of X | same class] • Mutual Information (originated by Shanon, 1948 and used for the measure of dependence by Perez, 1957, Russian) Darbellay (1999, CSDA) gives a good survey on the measure of statistical dependenceusing MI Variable Selection

25. Subset selection with DAVIS– ranking variables • MDI(meaured of departure from indep.) • ReliefF • MI (measure of Information) Variable Selection

26. Subset selection with DAVIS-decision trees • Discretization required Variable Selection

27. Subset Selection with DAVIS- stepwise discriminant analysis • Continuous variables only • Good under normality Variable Selection

28. Subset Selection with DAVIS- Mutual Information • Conventional approach: Discretization required • Normal mixture approach: Good for continuous variables • Incremental Algorith: Good for complex data Variable Selection

29. Variable Selection with DAVIS-design layout Variable Selection

30. Variable selection –titanic data • Variable ranking: sex, class, age • subset selection: age, class Variable Selection

31. Concluding remarks • DAVIS is a Java-based system • Any statistical model can be added to the system as a visual component if it follows certain rules. • Need more efficient design layout for various strategies of variable selection. • Need to coin easier-to-understand terminologies for various elements of the component. Variable Selection