advanced data mining introduction n.
Skip this Video
Loading SlideShow in 5 Seconds..
Advanced Data Mining: Introduction PowerPoint Presentation
Download Presentation
Advanced Data Mining: Introduction

Loading in 2 Seconds...

play fullscreen
1 / 17

Advanced Data Mining: Introduction - PowerPoint PPT Presentation

  • Uploaded on

Advanced Data Mining: Introduction. pms. Material Covered. Chapter 1 from Ullman’s book. Many slides are from the “Data Mining: Concepts and Techniques” book. Why Data Mining?. The Explosive Growth of Data: from terabytes to petabytes

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Advanced Data Mining: Introduction

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
advanced data mining introduction

Advanced Data Mining:Introduction

material covered
Material Covered
  • Chapter 1 from Ullman’s book.
  • Many slides are from the “Data Mining: Concepts and Techniques” book.
why data mining
Why Data Mining?
  • The Explosive Growth of Data: from terabytes to petabytes
    • Data collection and data availability
      • Automated data collection tools, database systems, Web, computerized society
    • Major sources of abundant data
      • Business: Web, e-commerce, transactions, stocks, …
      • Science: Remote sensing, bioinformatics, scientific simulation, …
      • Society and everyone: news, digital cameras, YouTube
  • We are drowning in data, but starving for knowledge!
  • “Necessity is the mother of invention”—Data mining—Automated analysis of massive data sets
evolution of sciences
Evolution of Sciences
  • Before 1600, empirical science
  • 1600-1950s, theoretical science
    • Each discipline has grown a theoretical component. Theoretical models often motivate experiments and generalize our understanding.
  • 1950s-1990s, computational science
    • Over the last 50 years, most disciplines have grown a third, computational branch (e.g. empirical, theoretical, and computational ecology, or physics, or linguistics.)
    • Computational Science traditionally meant simulation. It grew out of our inability to find closed-form solutions for complex mathematical models.
  • 1990-now, data science
    • The flood of data from new scientific instruments and simulations
    • The ability to economically store and manage petabytes of data online
    • The Internet and computing Grid that makes all these archives universally accessible
    • Scientific info. management, acquisition, organization, query, and visualization tasks scale almost linearly with data volumes. Data mining is a major new challenge!
  • Jim Gray and Alex Szalay, The World Wide Telescope: An Archetype for Online Science, Comm. ACM, 45(11): 50-54, Nov. 2002

from “Data Mining: Concepts and Techniques”

what is data mining
What Is Data Mining?
  • Data mining (knowledge discovery from data)
    • Extraction of interesting (non-trivial,implicit, previously unknown and potentially useful)patterns or knowledge from huge amount of data
    • Data mining: a misnomer?
  • Alternative names
    • Knowledge discovery (mining) in databases (KDD), knowledge extraction, data/pattern analysis, business intelligence, etc.
  • Watch out: Is everything “data mining”?
  • Negative examples:
    • Simple search and query processing
    • (Deductive) expert systems

from “Data Mining: Concepts and Techniques”

knowledge discovery kdd process
Knowledge Discovery (KDD) Process


  • This is a view from typical database systems and data warehousing communities
  • Data mining plays an essential role in the knowledge discovery process

Pattern Evaluation

Data Mining

Task-relevant Data


Data Warehouse

Data Cleaning

Data Integration


from “Data Mining: Concepts and Techniques”

data mining in business intelligence
Data Mining in Business Intelligence

Increasing potential

to support

business decisions

End User




Data Presentation

Visualization Techniques

Data Mining



Information Discovery

Data Exploration

Statistical Summary, Querying, and Reporting

Data Preprocessing/Integration, Data Warehouses


Data Sources

Paper, Files, Web documents, Scientific experiments, Database Systems

from “Data Mining: Concepts and Techniques”

directions in modeling
Directions in modeling
  • Pattern extraction  Model Discovery
  • Statistical modeling
    • E.g., decide that the data comes from a Gaussian distribution, estimate μ,σ parameters.
  • Machine learning
    • Train an algorithm, then apply to new data.
  • Results of Complex Queries (computational approaches)
    • E.g., summarization of the importance of a webpage in the form of a “pagerank” value.
    • E.g., prominent feature extraction, such as frequent itemsets and similar items.
multi dimensional view of data mining
Multi-Dimensional View of Data Mining
  • Knowledge to be mined (or: Data mining functions)
    • Characterization, discrimination, association, classification, clustering, trend/deviation, outlier analysis, etc.
    • Descriptive vs. predictive data mining
    • Multiple/integrated functions and mining at multiple levels
  • Data to be mined
    • Database data (extended-relational, object-oriented, heterogeneous, legacy), data warehouse, transactional data, stream, spatiotemporal, time-series, sequence, text and web, multi-media, graphs & social and information networks
  • Techniques utilized
    • Data-intensive, data warehouse (OLAP), machine learning, statistics, pattern recognition, visualization, high-performance, etc.
  • Applications adapted
    • Retail, telecommunication, banking, fraud analysis, bio-data mining, stock market analysis, text mining, Web mining, etc.

from “Data Mining: Concepts and Techniques”

meaningfulness of patterns
Meaningfulness of patterns
  • A big data-mining risk is that you will “discover” patterns that are meaningless.
  • Bonferroni’s principle: (roughly) if you look in more places for interesting patterns than your amount of data will support, you are bound to find meaningless patterns
rhine paradox
Rhine Paradox
  • Joseph Rhine was a parapsychologist in the 1950’s who hypothesized that some people had Extra-Sensory Perception
  • He devised an experiment where subjects were asked to guess 10 hidden cards – red or blue
  • He discovered that almost 1 in 1000 had ESP – they were able to get all 10 right!
  • He told these people they had ESP and called them in for another test of the same type
  • Alas, he discovered that almost all of them had lost their ESP
  • What did he conclude?
  • He concluded that you shouldn’t tell people they have ESP; it causes them to lose it!
major challenges in data mining
Major Challenges in Data Mining
  • Efficiency and scalability of data mining algorithms
  • Parallel, distributed, stream, and incremental mining methods
  • Handling high-dimensionality
  • Handling noise, uncertainty, and incompleteness of data
  • Incorporation of constraints, expert knowledge, and background knowledge in data mining
  • Pattern evaluation and knowledge integration
  • Mining diverse and heterogeneous kinds of data: e.g., bioinformatics, Web, software/system engineering, information networks
  • Application-oriented and domain-specific data mining
  • Invisible data mining (embedded in other functional modules)
  • Protection of security, integrity, and privacy in data mining

from “Data Mining: Concepts and Techniques”

things useful to know
Things Useful to Know
  • Probability
  • Linear Algebra basics
  • Hash functions
  • Indices
  • Secondary storage
  • Power laws
big data
Big Data


  • Tiny  0s
  • Small  1000s fitting in memory
  • Medium  1000000 (may not fit in memory)
  • Large  1000000000
  • Huge  1000000000000 ++

From Graefe’s “New algorithms for join and grouping operations”, 2011.