Data Preprocessing - PowerPoint PPT Presentation

Data preprocessing
1 / 29

  • Updated On :
  • Presentation posted in: General

Data Preprocessing. Dr. Bernard Chen Ph.D. University of Central Arkansas Fall 2009. Knowledge Discovery (KDD) Process. Knowledge. Pattern Evaluation. Data mining—core of knowledge discovery process. Data Mining. Task-relevant Data. Selection. Data Warehouse. Data Cleaning.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Data Preprocessing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Data preprocessing

Data Preprocessing

Dr. Bernard Chen Ph.D.

University of Central Arkansas

Fall 2009

Knowledge discovery kdd process

Knowledge Discovery (KDD) Process


Pattern Evaluation

  • Data mining—core of knowledge discovery process

Data Mining

Task-relevant Data


Data Warehouse

Data Cleaning

Data Integration


Knowledge process

Knowledge Process

  • Data cleaning – to remove noise and inconsistent data

  • Data integration – to combine multiple source

  • Data selection – to retrieve relevant data for analysis

  • Data transformation – to transform data into appropriate form for data mining

  • Data mining

  • Evaluation

  • Knowledge presentation

Why preprocess the data

Why Preprocess the data

  • Image that you are a manager at ALLElectronics and have been charger with analyzing the company’s data

  • Then you realize:

    • Several of the attributes for carious tuples have no recorded value

    • Some information you want is not on recorded

    • Some values are reported as incomplete, noisy, and inconsistent

  • Welcome to real world!!

Why data preprocessing

Why Data Preprocessing?

  • Data in the real world is dirty

    • incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data

      • e.g., occupation=“ ”

    • noisy: containing errors or outliers

      • e.g., Salary=“-10”

    • inconsistent: containing discrepancies in codes or names

      • e.g., Age=“42” Birthday=“03/07/1997”

      • e.g., Was rating “1,2,3”, now rating “A, B, C”

      • e.g., discrepancy between duplicate records

Why is data dirty

Why Is Data Dirty?

  • Incomplete data may come from

    • “Not applicable” data value when collected

    • Different considerations between the time when the data was collected and when it is analyzed.

    • Human/hardware/software problems

Why is data dirty1

Why Is Data Dirty?

  • Noisy data (incorrect values) may come from

    • Faulty data collection instruments

    • Human or computer error at data entry

    • Errors in data transmission

Why is data dirty2

Why Is Data Dirty?

  • Inconsistent data may come from

    • Different data sources

    • Functional dependency violation (e.g., modify some linked data)

  • Duplicate records also need data cleaning

Why is data preprocessing important

Why Is Data Preprocessing Important?

  • No quality data, no quality mining results!

    • Quality decisions must be based on quality data

    • e.g., duplicate or missing data may cause incorrect or even misleading statistics.

  • Data extraction, cleaning, and transformation comprises the majority of the work of building a data warehouse

Major tasks in data preprocessing

Major Tasks in Data Preprocessing

  • Data cleaning

    • Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies

  • Data integration

    • Integration of multiple databases, data cubes, or files

  • Data transformation

    • Normalization and aggregation

Major tasks in data preprocessing1

Major Tasks in Data Preprocessing

  • Data reduction

    • Obtains reduced representation in volume but produces the same or similar analytical results

  • Data discretization

    • Part of data reduction but with particular importance, especially for numerical data

Forms of data preprocessing

Forms of Data Preprocessing

Descriptive data summarization

Descriptive data summarization

  • Motivation

    • To better understand the data: central tendency, variation and spread

  • Data dispersion characteristics

    • median, max, min, quantiles, outliers, variance, etc.

Descriptive data summarization1

Descriptive data summarization

  • Numerical dimensions correspond to sorted intervals

    • Data dispersion: analyzed with multiple granularities of precision

    • Boxplot or quantile analysis on sorted intervals

Measuring the central tendency

Measuring the Central Tendency

  • Mean

  • Median

  • Mode

    • Value that occurs most frequently in the data

    • Dataset with one, two or three modes are respectively called unimodal, bimodal, and trimodal

Symmetric vs skewed data

Symmetric vs. Skewed Data

Measuring the central tendency1

Measuring the Central Tendency

  • For unimodal frequency curves that are moderately skewed, we have the following empirical relation:

    mean – mode = 3 * (mean - median)

Measuring the dispersion of data

Measuring the Dispersion of Data

  • Quartiles, outliers and boxplots

    • The median is the 50th percentile

    • Quartiles: Q1 (25th percentile), Q3 (75th percentile)

    • Inter-quartile range: IQR = Q3 –Q1

    • Outlier: usually, a value higher/lower than 1.5 x IQR

Boxplot analysis

Boxplot Analysis

  • Five-number summary of a distribution:

    Minimum, Q1, M, Q3, Maximum

  • Boxplot

    • Data is represented with a box

    • The ends of the box are at the first and third quartiles, i.e., the height of the box is IRQ

    • The median is marked by a line within the box

    • Whiskers: two lines outside the box extend to Minimum and Maximum

Boxplot analysis1

Boxplot Analysis

Histogram analysis

Histogram Analysis

  • Graph displays of basic statistical class descriptions

    • Frequency histograms

      • A univariate graphical method

      • Consists of a set of rectangles that reflect the counts or frequencies of the classes present in the given data

Histogram analysis1

Histogram Analysis

Quantile plot

Quantile Plot

  • Displays all of the data (allowing the user to assess both the overall behavior and unusual occurrences)

  • Plots quantile information

    • For a data xidata sorted in increasing order, fiindicates that approximately 100 fi% of the data are below or equal to the value xi

Quantile plot1

Quantile Plot

Quantile quantile q q plot

Quantile-Quantile (Q-Q) Plot

  • Graphs the quantiles of one univariate distribution against the corresponding quantiles of another

  • Allows the user to view whether there is a shift in going from one distribution to another

Quantile quantile q q plot1

Quantile-Quantile (Q-Q) Plot

Scatter plot

Scatter plot

  • Provides a first look at bivariate data to see clusters of points, outliers, etc

  • Each pair of values is treated as a pair of coordinates and plotted as points in the plane

Scatter plot1

Scatter plot

Loess curve

Loess Curve

  • Login