data mining and decision trees n.
Skip this Video
Loading SlideShow in 5 Seconds..
Data Mining and Decision Trees PowerPoint Presentation
Download Presentation
Data Mining and Decision Trees

Loading in 2 Seconds...

play fullscreen
1 / 66

Data Mining and Decision Trees - PowerPoint PPT Presentation

  • Uploaded on

Lecture 2 CS157B. Data Mining and Decision Trees. Prof. Sin-Min Lee Department of Computer Science. Evolution of Database Technology. 1960s: Data collection, database creation, IMS and network DBMS 1970s: Relational data model, relational DBMS implementation 1980s:

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Data Mining and Decision Trees' - doctor

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
data mining and decision trees

Lecture 2


Data Mining and Decision Trees

Prof. Sin-Min Lee

Department of Computer Science

evolution of database technology
Evolution of Database Technology
  • 1960s:
    • Data collection, database creation, IMS and network DBMS
  • 1970s:
    • Relational data model, relational DBMS implementation
  • 1980s:
    • RDBMS, advanced data models (extended-relational, OO, deductive, etc.) and application-oriented DBMS (spatial, scientific, engineering, etc.)
  • 1990s—2000s:
    • Data mining and data warehousing, multimedia databases, and Web databases
what is data mining
What Is Data Mining?
  • Data mining (knowledge discovery in databases):
    • Extraction of interesting (non-trivial,implicit, previously unknown and potentially useful)information or patterns from data in large databases
  • Alternative names and their “inside stories”:
    • Data mining: a misnomer?
    • Knowledge discovery(mining) in databases (KDD), knowledge extraction, data/pattern analysis, data archeology, data dredging, information harvesting, business intelligence, etc.
  • What is not data mining?
    • (Deductive) query processing.
    • Expert systems or small ML/statistical programs
why data mining potential applications
Why Data Mining? — Potential Applications
  • Database analysis and decision support
    • Market analysis and management
      • target marketing, customer relation management, market basket analysis, cross selling, market segmentation
    • Risk analysis and management
      • Forecasting, customer retention, improved underwriting, quality control, competitive analysis
    • Fraud detection and management
  • Other Applications
    • Text mining (news group, email, documents) and Web analysis.
    • Intelligent query answering
market analysis and management 1
Market Analysis and Management (1)
  • Where are the data sources for analysis?
    • Credit card transactions, loyalty cards, discount coupons, customer complaint calls, plus (public) lifestyle studies
  • Target marketing
    • Find clusters of “model” customers who share the same characteristics: interest, income level, spending habits, etc.
  • Determine customer purchasing patterns over time
    • Conversion of single to a joint bank account: marriage, etc.
  • Cross-market analysis
    • Associations/co-relations between product sales
    • Prediction based on the association information
market analysis and management 2
Market Analysis and Management (2)
  • Customer profiling
    • data mining can tell you what types of customers buy what products (clustering or classification)
  • Identifying customer requirements
    • identifying the best products for different customers
    • use prediction to find what factors will attract new customers
  • Provides summary information
    • various multidimensional summary reports
    • statistical summary information (data central tendency and variation)
corporate analysis and risk management
Corporate Analysis and Risk Management
  • Finance planning and asset evaluation
    • cash flow analysis and prediction
    • contingent claim analysis to evaluate assets
    • cross-sectional and time series analysis (financial-ratio, trend analysis, etc.)
  • Resource planning:
    • summarize and compare the resources and spending
  • Competition:
    • monitor competitors and market directions
    • group customers into classes and a class-based pricing procedure
    • set pricing strategy in a highly competitive market
fraud detection and management 1
Fraud Detection and Management (1)
  • Applications
    • widely used in health care, retail, credit card services, telecommunications (phone card fraud), etc.
  • Approach
    • use historical data to build models of fraudulent behavior and use data mining to help identify similar instances
  • Examples
    • auto insurance: detect a group of people who stage accidents to collect on insurance
    • money laundering: detect suspicious money transactions (US Treasury's Financial Crimes Enforcement Network)
    • medical insurance: detect professional patients and ring of doctors and ring of references
fraud detection and management 2
Fraud Detection and Management (2)
  • Detecting inappropriate medical treatment
    • Australian Health Insurance Commission identifies that in many cases blanket screening tests were requested (save Australian $1m/yr).
  • Detecting telephone fraud
    • Telephone call model: destination of the call, duration, time of day or week. Analyze patterns that deviate from an expected norm.
    • British Telecom identified discrete groups of callers with frequent intra-group calls, especially mobile phones, and broke a multimillion dollar fraud.
  • Retail
    • Analysts estimate that 38% of retail shrink is due to dishonest employees.
other applications
Other Applications
  • Sports
    • IBM Advanced Scout analyzed NBA game statistics (shots blocked, assists, and fouls) to gain competitive advantage for New York Knicks and Miami Heat
  • Astronomy
    • JPL and the Palomar Observatory discovered 22 quasars with the help of data mining
  • Internet Web Surf-Aid
    • IBM Surf-Aid applies data mining algorithms to Web access logs for market-related pages to discover customer preference and behavior pages, analyzing effectiveness of Web marketing, improving Web site organization, etc.
data mining a kdd process
Data Mining: A KDD Process


Pattern Evaluation

  • Data mining: the core of knowledge discovery process.

Data Mining

Task-relevant Data


Data Warehouse

Data Cleaning

Data Integration


steps of a kdd process
Steps of a KDD Process
  • Learning the application domain:
    • relevant prior knowledge and goals of application
  • Creating a target data set: data selection
  • Data cleaning and preprocessing: (may take 60% of effort!)
  • Data reduction and transformation:
    • Find useful features, dimensionality/variable reduction, invariant representation.
  • Choosing functions of data mining
    • summarization, classification, regression, association, clustering.
  • Choosing the mining algorithm(s)
  • Data mining: search for patterns of interest
  • Pattern evaluation and knowledge presentation
    • visualization, transformation, removing redundant patterns, etc.
  • Use of discovered knowledge
area 1 risk analysis
Area 1: Risk Analysis
  • Insurance companies and banks use data mining for risk analysis.
  • And insurance company searches in its own insurants and claims databases for relationships between personal characteristics and claim behavior.
  • The company is especially interested in the characteristics of insurants with a high deviating claim behavior.
  • With data mining, these so-called risk-profiles can be discovered and the company can use this information to adapt its premium polity.
area 2 direct marketing
Area 2: Direct Marketing
  • Data mining can also be used to discover the relationship between one’s personal characteristics, e.g. age, gender, hometown, and the probability that one will respond to a mailing.
  • Such relationships can be used to select those customers from the mailing database that have the highest probability of responding to a mailing.
This allows the company to mail its prospects selectively, thus maximizing the response.
  • For example:

1. Company X sends a mailing to a number of prospects.

2. The response is 2%.

what data mining can do
What Data Mining can do
  • Enables companies to determine relationships among “internal” and “external” factors.
  • Predict cross-sell opportunities and make recommendations
  • Segment markets and personalize communications.
  • Predicts outcomes of future situations
the process of data mining
The process Of Data Mining
  • There are 3 main steps in the Data Mining process:
    • Preparation: data is selected from the warehouse and “cleansed”.
    • Processing: algorithms are used to process the data. This step uses modeling to make predictions.
    • Analysis: output is evaluated.
reasons for growing popularity
Reasons for growing popularity
  • Growing data volume- enormous amount of existing and appearing data that require processing.
  • Limitations of Human Analysis- humans lacking objectiveness when analyzing dependencies for data.
  • Low cost of Machine Learning- the data mining process has a lower cost than hiring highly trained professionals to analyze data.
data mining techniques
Data Mining Techniques
  • Association Rule- is to discover interesting associations between attributes that are contained in a database.
  • Clustering- finds appropriate groupings of elements for a set of data.
  • Sequential patterns-looking for patterns where one event leads to another later event.
  • Classification- looking for new patterns.
applications of data mining
Applications of Data Mining
  • Data Mining is applied in the following areas:
    • Prediction of the Stock Market: predicting the future trends.
    • Bankruptcy prediction: prediction based on computer generated rules, using models
    • Foreign Exchange Market: Data Mining is used to identify trading rules.
    • Fraud Detection: construction of algorithms and models that will help recognize a variety of fraud patterns.
results of data mining include
Results of Data Mining Include:
  • Forecasting what may happen in the future
  • Classifying people or things into groups by recognizing patterns
  • Clustering people or things into groups based on their attributes
  • Associating what events are likely to occur together
  • Sequencing what events are likely to lead to later events
data mining is not
Data mining is not
  • Brute-force crunching of bulk data
  • “Blind” application of algorithms
  • Going to find relationships where none exist
  • Presenting data in different ways
  • A database intensive task
  • A difficult to understand technology requiring an advanced degree in computer science
what data mining has done for
What data mining has done for...

The US Internal Revenue Service

needed to improve customer service and...

Scheduled its workforce

to provide faster, more accurate answers to questions.

what data mining has done for1
What data mining has done for...

The US Drug Enforcement Agency needed to be more effective in their drug “busts” and

analyzed suspects’ cell phone usage to focus investigations.

what data mining has done for2
What data mining has done for...

HSBC need to cross-sell more

effectively by identifying profiles

that would be interested in higher

yielding investments and...

Reduced direct mail costs by 30% while garnering 95% of the campaign’s revenue.


Decision Trees

  • A decision tree is a special case of a state-space graph.
  • It is a rooted tree in which each internal node corresponds to a decision, with a subtree at these nodes for each possible outcome of the decision.
  • Decision trees can be used to model problems in which a series of decisions leads to a solution.
  • The possible solutions of the problem correspond to the paths from the root to the leaves of the decision tree.

Decision Trees

  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • Q
  • Q
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • x
  • Example: The n-queens problem
  • How can we place n queens on an nn chessboard so that no two queens can capture each other?

A queen can move any number of squares horizontally, vertically, and diagonally.

Here, the possible target squares of the queen Q are marked with an x.


Let us consider the 4-queens problem.

  • Question: How many possible configurations of 44 chessboards containing 4 queens are there?
  • Answer: There are 16!/(12!4!) = (13141516)/(234) = 13754 = 1820 possible configurations.
  • Shall we simply try them out one by one until we encounter a solution?
  • No, it is generally useful to think about a search problem more carefully and discover constraints on the problem’s solutions.
  • Such constraints can dramatically reduce the size of the relevant state space.

Obviously, in any solution of the n-queens problem, there must be exactly one queen in each column of the board.

Otherwise, the two queens in the same column could capture each other.

Therefore, we can describe the solution of this problem as a sequence of n decisions:

Decision 1: Place a queen in the first column.

Decision 2: Place a queen in the second column.

...Decision n: Place a queen in the n-th column.


Backtracking in Decision Trees

  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q
  • Q

empty board

place 1st queen

place 2nd queen

place 3rd queen

place 4th queen


Neural Network

Many inputs and a single output

Trained on signal and background sample

Well understood and mostly accepted in HEP

Decision Tree

Many inputs and a single output

Trained on signal and background sample

Used mostly in life sciences & business

decision tree basic algorithm
Decision treeBasic Algorithm
  • Initialize top node to all examples
  • While impure leaves available
    • select next impure leave L
    • find splitting attribute A with maximal information gain
    • for each value of A add child to L
decision tree find good split

gain: 0.25 bits


gain: 0.16 bits


gain: 0.03 bits


gain: 0.14 bits


Decision treeFind good split
  • Sufficient statistics to compute info gain: count matrix
decision trees
Decision trees
  • Simple depth-first construction
  • Needs entire data to fit in memory
  • Unsuitable for large data sets
  • Need to “scale up”
decision trees1
Decision Trees
  • Enable a business to quantify decision making
  • Useful when the outcomes are uncertain
  • Places a numerical value on likely or potential outcomes
  • Allows comparison of different possible decisions to be made
decision trees2
Decision Trees
  • Limitations:
    • How accurate is the data used in the construction of the tree?
    • How reliable are the estimates of the probabilities?
    • Data may be historical – does this data relate to real time?
    • Necessity of factoring in the qualitative factors – human resources, motivation, reaction, relations with suppliers and other stakeholders
the process
The Process

Economic growth rises

Expected outcome£300,000


Expand by opening new outlet

Economic growth declines

Expected outcome



Maintain current status


The circle denotes the point where different outcomes could occur. The estimates of the probability and the knowledge of the expected outcome allow the firm to make a calculation of the likely return. In this example it is:

Economic growth rises: 0.7 x £300,000 = £210,000

Economic growth declines: 0.3 x £500,000 = -£150,000

The calculation would suggest it is wise to go ahead with the decision ( a net ‘benefit’ figure of +£60,000)

A square denotes the point where a decision is made, In this example, a business is contemplating opening a new outlet. The uncertainty is the state of the economy – if the economy continues to grow healthily the option is estimated to yield profits of £300,000. However, if the economy fails to grow as expected, the potential loss is estimated at £500,000.

There is also the option to do nothing and maintain the current status quo! This would have an outcome of £0.


The Process

Economic growth rises

Expected outcome£300,000


Expand by opening new outlet

Economic growth declines

Expected outcome



Maintain current status


Look what happens however if the probabilities change. If the firm is unsure of the potential for growth, it might estimate it at 50:50. In this case the outcomes will be:

Economic growth rises: 0.5 x £300,000 = £150,000

Economic growth declines: 0.5 x -£500,000 = -£250,000

In this instance, the net benefit is -£100,000 – the decision looks less favourable!


Trained Decision Tree


(Binned Likelihood Fit)

decision trees from data base
Decision Trees from Data Base

Ex Att Att Att Concept

Num Size Colour Shape Satisfied

1 med blue brick yes

2 small red wedge no

3 small red sphere yes

4 large red wedge no

5 large green pillar yes

6 large red pillar no

7 large green sphere yes

Choose target : Concept satisfied

Use all attributes except Ex Num

rules from tree
Rules from Tree

IF (SIZE = large


((SHAPE = wedge) OR (SHAPE = pillar AND

COLOUR = red) )))

OR (SIZE = small AND SHAPE = wedge)


IF (SIZE = large


((SHAPE = pillar) AND COLOUR = green)

OR SHAPE = sphere) )

OR (SIZE = small AND SHAPE = sphere)

OR (SIZE = medium)


disjunctive normal form dnf
Disjunctive Normal Form - DNF


(SIZE = medium)


(SIZE = small AND SHAPE = sphere)


(SIZE = large AND SHAPE = sphere)


(SIZE = large AND SHAPE = pillar

AND COLOUR = green

THEN CONCEPT = satisfied

ELSE CIONCEPT = not satisfied