Data PreProcessing

1. Data PreProcessing

2. Data Preprocessing • Why preprocess the data? • Descriptive data summarization • Data cleaning • Data integration and transformation • Data reduction • Discretization and concept hierarchy generation

3. Why Data Preprocessing? • Data in the real world is dirty • incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data • e.g., occupation=“ ” • noisy: containing errors or outliers • e.g., Salary=“-10” • inconsistent: containing discrepancies in codes or names • e.g., Age=“42” Birthday=“03/07/1997” • e.g., Was rating “1,2,3”, now rating “A, B, C” • e.g., discrepancy between duplicate records

4. Why Is Data Dirty? • Incomplete data may come from • “Not applicable” data value when collected • Different considerations between the time when the data was collected and when it is analyzed. • Human/hardware/software problems • Noisy data (incorrect values) may come from • Faulty data collection instruments • Human or computer error at data entry • Errors in data transmission • Inconsistent data may come from • Different data sources • Functional dependency violation (e.g., modify some linked data) • Duplicate records also need data cleaning

5. Data Quality • What kinds of data quality problems? • How can we detect problems with the data? • What can we do about these problems? • Examples of data quality problems: • Noise and outliers • missing values • duplicate data

6. Noise • Noise refers to modification of original values • Examples: distortion of a person’s voice when talking on a poor phone and “snow” on television screen Two Sine Waves Two Sine Waves + Noise

7. Outliers • Outliers are data objects with characteristics that are considerably different than most of the other data objects in the data set

8. Missing Values • Reasons for missing values • Information is not collected (e.g., people decline to give their age and weight) • Attributes may not be applicable to all cases (e.g., annual income is not applicable to children) • Handling missing values • Eliminate Data Objects • Estimate Missing Values • Ignore the Missing Value During Analysis • Replace with all possible values (weighted by their probabilities)

9. Duplicate Data • Data set may include data objects that are duplicates, or almost duplicates of one another • Major issue when merging data from heterogeous sources • Examples: • Same person with multiple email addresses • Data cleaning • Process of dealing with duplicate data issues

10. Why Is Data Preprocessing Important? • No quality data, no quality mining results! • Quality decisions must be based on quality data • e.g., duplicate or missing data may cause incorrect or even misleading statistics. • Data warehouse needs consistent integration of quality data • Data extraction, cleaning, and transformation comprises the majority of the work of building a data warehouse

11. Multi-Dimensional Measure of Data Quality • A well-accepted multidimensional view: • Accuracy • Completeness • Consistency • Timeliness • Believability • Value added • Interpretability • Accessibility • Broad categories: • Intrinsic, contextual, representational, and accessibility

12. Major Tasks in Data Preprocessing • Data cleaning • Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies • Data integration • Integration of multiple databases, data cubes, or files • Data transformation • Normalization and aggregation • Data reduction • Obtains reduced representation in volume but produces the same or similar analytical results • Data discretization • Part of data reduction but with particular importance, especially for numerical data

13. Forms of Data Preprocessing

14. Types of Data Sets • Record • Relational records • Data matrix, e.g., numerical matrix, crosstabs • Document data: text documents: term-frequency vector • Transaction data • Graph and network • World Wide Web • Social or information networks • Molecular Structures • Ordered • Video data: sequence of images • Temporal data: time-series • Sequential Data: transaction sequences • Genetic sequence data • Spatial, image and multimedia: • Spatial data: maps • Image data: • Video data:

15. Important Characteristics of Structured Data • Dimensionality • Curse of dimensionality • Sparsity • Only presence counts • Resolution • Patterns depend on the scale • Distribution • Centrality and dispersion

16. Record Data • Data that consists of a collection of records, each of which consists of a fixed set of attributes

17. Data Matrix • If data objects have the same fixed set of numeric attributes, then the data objects can be thought of as points in a multi-dimensional space, where each dimension represents a distinct attribute • Such data set can be represented by an m by n matrix, where there are m rows, one for each object, and n columns, one for each attribute

18. Document Data • Each document becomes a `term' vector, • each term is a component (attribute) of the vector, • the value of each component is the number of times the corresponding term occurs in the document.

19. Transaction Data • A special type of record data, where • each record (transaction) involves a set of items. • For example, consider a grocery store. The set of products purchased by a customer during one shopping trip constitute a transaction, while the individual products that were purchased are the items.

20. Graph Data • Examples: Generic graph and HTML Links

21. Chemical Data • Benzene Molecule: C6H6

22. Ordered Data • Sequences of transactions Items/Events An element of the sequence

23. Ordered Data • Genomic sequence data

24. Ordered Data • Spatio-Temporal Data Average Monthly Temperature of land and ocean

25. Data Objects • Data sets are made up of data objects. • A data object represents an entity. • Examples: • sales database: customers, store items, sales • medical database: patients, treatments • university database: students, professors, courses • Also called samples , examples, instances, data points, objects, tuples. • Data objects are described by attributes. • Database rows -> data objects; columns ->attributes.

26. Attributes • Attribute (or dimensions, features, variables): a data field, representing a characteristic or feature of a data object. • E.g., customer _ID, name, address • Types: • Nominal • Binary • Numeric: quantitative • Interval-scaled • Ratio-scaled

27. Attribute Types • Nominal: categories, states, or “names of things” • Hair_color = {auburn, black, blond, brown, grey, red, white} • occupation, ID numbers, zip codes • Binary • Nominal attribute with only 2 states (0 and 1) • Symmetric binary: both outcomes equally important • e.g., employment status • Asymmetric binary: outcomes not equally important. • e.g., medical test (positive vs. negative) • Convention: assign 1 to most important outcome • Ordinal • Values have a meaningful order (ranking) but magnitude between successive values is not known. • Size = {small, medium, large}, grades, army rankings

28. Numeric Attribute Types • Quantity (integer or real-valued) • Interval • Measured on a scale of equal-sized units • Values have order • E.g., temperature in C˚or F˚, calendar dates • No true zero-point • Ratio • Inherent zero-point • We can speak of values as being an order of magnitude larger than the unit of measurement (10 K˚ is twice as high as 5 K˚). • e.g., temperature in Kelvin, length, counts, monetary quantities

29. Properties of Attribute Values • The type of an attribute depends on which of the following properties it possesses: • Distinctness: =  • Order: < > • Addition: + - • Multiplication: * / • Nominal attribute: distinctness • Ordinal attribute: distinctness & order • Interval attribute: distinctness, order & addition • Ratio attribute: all 4 properties

30. Attribute Type Description Examples Operations Nominal The values of a nominal attribute are just different names, i.e., nominal attributes provide only enough information to distinguish one object from another. (=, ) zip codes, employee ID numbers, eye color, sex: {male, female} mode, entropy, contingency correlation, 2 test Ordinal The values of an ordinal attribute provide enough information to order objects. (<, >) hardness of minerals, {good, better, best}, grades, street numbers median, percentiles, rank correlation, run tests, sign tests Interval For interval attributes, the differences between values are meaningful, i.e., a unit of measurement exists. (+, - ) calendar dates, temperature in Celsius or Fahrenheit mean, standard deviation, Pearson's correlation, t and F tests Ratio For ratio variables, both differences and ratios are meaningful. (*, /) temperature in Kelvin, monetary quantities, counts, age, mass, length, electrical current geometric mean, harmonic mean, percent variation

31. Attribute Level Transformation Comments Nominal Any permutation of values If all employee ID numbers were reassigned, would it make any difference? Ordinal An order preserving change of values, i.e., new_value = f(old_value) where f is a monotonic function. An attribute encompassing the notion of good, better best can be represented equally well by the values {1, 2, 3} or by { 0.5, 1, 10}. Interval new_value =a * old_value + b where a and b are constants Thus, the Fahrenheit and Celsius temperature scales differ in terms of where their zero value is and the size of a unit (degree). Ratio new_value = a * old_value Length can be measured in meters or feet.

32. Discrete vs. Continuous Attributes • DiscreteAttribute • Has only a finite or countably infinite set of values • E.g., zip codes, profession, or the set of words in a collection of documents • Sometimes, represented as integer variables • Note: Binary attributes are a special case of discrete attributes • ContinuousAttribute • Has real numbers as attribute values • E.g., temperature, height, or weight • Practically, real values can only be measured and represented using a finite number of digits • Continuous attributes are typically represented as floating-point variables

33. Mining Data DescriptiveCharacteristics • Motivation • To better understand the data: central tendency, variation and spread • Data dispersion characteristics • median, max, min, quantiles, outliers, variance, etc. • Numerical dimensions correspond to sorted intervals • Data dispersion: analyzed with multiple granularities of precision • Boxplot or quantile analysis on sorted intervals • Dispersion analysis on computed measures • Folding measures into numerical dimensions • Boxplot or quantile analysis on the transformed cube

34. Summary Statistics • Summary statistics are numbers that summarize properties of the data • Summarized properties include frequency, location and spread • Examples: location - mean spread - standard deviation • Most summary statistics can be calculated in a single pass through the data

35. Measures of Location: Mean and Median • The mean is the most common measure of the location of a set of points. • However, the mean is very sensitive to outliers. • Thus, the median or a trimmed mean is also commonly used.

36. Measuring the Central Tendency • Mean (algebraic measure) (sample vs. population): • Weighted arithmetic mean: • Trimmed mean: chopping extreme values • Median: A holistic measure • Middle value if odd number of values, or average of the middle two values otherwise • Estimated by interpolation (for grouped data): • Mode • Value that occurs most frequently in the data • Unimodal, bimodal, trimodal • Empirical formula:

37. Symmetric vs. Skewed Data • Median, mean and mode of symmetric, positively and negatively skewed data

38. Percentiles • For continuous data, the notion of a percentile is more useful. Given an ordinal or continuous attribute x and a number p between 0 and 100, the pth percentile is a value of x such that p% of the observed values of x are less than . • For instance, the 50th percentile is the value such that 50% of all values of x are less than .

39. Measuring the Dispersion of Data • Quartiles, outliers and boxplots • Quartiles: Q1 (25th percentile), Q3 (75th percentile) • Inter-quartile range: IQR = Q3 –Q1 • Five number summary: min, Q1, M,Q3, max • Boxplot: ends of the box are the quartiles, median is marked, whiskers, and plot outlier individually • Outlier: usually, a value higher/lower than 1.5 x IQR • Variance and standard deviation (sample:s, population: σ) • Variance: (algebraic, scalable computation) • Standard deviations (or σ) is the square root of variance s2 ( σ2)

40. Measures of Spread: Range and Variance • Range is the difference between the max and min • The variance or standard deviation is the most common measure of the spread of a set of points. • However, this is also sensitive to outliers, so that other measures are often used.

41. Properties of Normal Distribution Curve • The normal (distribution) curve • From μ–σ to μ+σ: contains about 68% of the measurements (μ: mean, σ: standard deviation) • From μ–2σ to μ+2σ: contains about 95% of it • From μ–3σ to μ+3σ: contains about 99.7% of it

42. Visualization Visualization is the conversion of data into a visual or tabular format so that the characteristics of the data and the relationships among data items or attributes can be analyzed or reported. • Visualization of data is one of the most powerful and appealing techniques for data exploration. • Humans have a well developed ability to analyze large amounts of information that is presented visually • Can detect general patterns and trends • Can detect outliers and unusual patterns

43. Representation • Is the mapping of information to a visual format • Data objects, their attributes, and the relationships among data objects are translated into graphical elements such as points, lines, shapes, and colors. • Example: • Objects are often represented as points • Their attribute values can be represented as the position of the points or the characteristics of the points, e.g., color, size, and shape • If position is used, then the relationships of points, i.e., whether they form groups or a point is an outlier, is easily perceived.

44. Arrangement • Is the placement of visual elements within a display • Can make a large difference in how easy it is to understand the data • Example:

45. Selection • Is the elimination or the de-emphasis of certain objects and attributes • Selection may involve the chossing a subset of attributes • Dimensionality reduction is often used to reduce the number of dimensions to two or three • Alternatively, pairs of attributes can be considered • Selection may also involve choosing a subset of objects • A region of the screen can only show so many points • Can sample, but want to preserve points in sparse areas

46. outlier 75th percentile 50th percentile 25th percentile 10th percentile 10th percentile Boxplot Analysis • 5 number summary of a distribution: Minimum, Q1, M, Q3, Maximum • Boxplot • Data is represented with a box • The ends of the box are at the first and third quartiles, i.e., the height of the box is IRQ • The median is marked by a line within the box • Whiskers: two lines outside the box extend to Minimum and Maximum

47. Visualization of Data Dispersion: Boxplot Analysis

48. Histogram Analysis • Graph displays of basic statistical class descriptions • Frequency histograms • A univariate graphical method • Consists of a set of rectangles that reflect the counts or frequencies of the classes present in the given data

49. Quantile Plot • Displays all of the data (allowing the user to assess both the overall behavior and unusual occurrences) • Plots quantile information • For a data xidata sorted in increasing order, fiindicates that approximately 100 fi% of the data are below or equal to the value xi

50. Quantile-Quantile (Q-Q) Plot • Graphs the quantiles of one univariate distribution against the corresponding quantiles of another • Allows the user to view whether there is a shift in going from one distribution to another