1 / 56

DATA MINING

DATA MINING. LECTURE 1 INTRODUCTION TO DATA MINING. Data Mining Outline. Introduction Related Concepts Data Mining Techniques. Introduction Outline. Goal: Provide an overview of data mining. Define data mining Data mining vs. databases Basic data mining tasks Data mining development

tminer
Download Presentation

DATA MINING

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DATA MINING LECTURE 1 INTRODUCTION TO DATA MINING

  2. Data Mining Outline • Introduction • Related Concepts • Data Mining Techniques DATA MINING VESIT M.VIJAYALAKSHMI

  3. Introduction Outline Goal: Provide an overview of data mining. • Define data mining • Data mining vs. databases • Basic data mining tasks • Data mining development • Data mining issues DATA MINING VESIT M.VIJAYALAKSHMI

  4. Introduction • Data is growing at a phenomenal rate • Users expect more sophisticated information • How? UNCOVER HIDDEN INFORMATION DATA MINING DATA MINING VESIT M.VIJAYALAKSHMI

  5. Data Mining Definition • Finding hidden information in a huge store of data • Fit data to a model • Similar terms • Exploratory data analysis • Data driven discovery • Deductive learning DATA MINING VESIT M.VIJAYALAKSHMI

  6. What Is Data Mining? • Data mining (knowledge discovery in databases): • Extraction of interesting (non-trivial,implicit, previously unknown and potentially useful)information or patterns from data in large databases • Alternative names and their “inside stories”: • Knowledge discovery(mining) in databases (KDD), knowledge extraction, data/pattern analysis, data archeology, data dredging, information harvesting, business intelligence, etc. • What is not data mining? • (Deductive) query processing. • Expert systems or small ML/statistical programs DATA MINING VESIT M.VIJAYALAKSHMI

  7. Potential Applications • Market analysis and management • target marketing, CRM, market basket analysis, cross selling, market segmentation • Risk analysis and management • Forecasting, customer retention, quality control, competitive analysis • Fraud detection and management • Text mining (news group, email, documents) and Web analysis. • Intelligent query answering DATA MINING VESIT M.VIJAYALAKSHMI

  8. Market Analysis and Management (1) • Where are the data sources for analysis? • Credit card transactions, loyalty cards, discount coupons, customer complaint calls, • Target marketing (Find clusters of “model” customers who share the same characteristics: interest, income level, spending habits, etc.) • Determine customer purchasing patterns over time • Cross-market analysis • Associations/co-relations between product sales • Prediction based on the association information DATA MINING VESIT M.VIJAYALAKSHMI

  9. Market Analysis and Management (2) • Customer profiling • data mining can tell you what types of customers buy what products (clustering or classification) • Identifying customer requirements • identifying the best products for different customers • use prediction to find what factors will attract new customers • Provides summary information • various multidimensional summary reports • statistical summary information (data central tendency and variation) DATA MINING VESIT M.VIJAYALAKSHMI

  10. Fraud Detection and Management • Applications • widely used in health care, retail, credit card services, telecommunications (phone card fraud), etc. • Approach • use historical data to build models of fraudulent behavior and use data mining to help identify similar instances • Examples • auto insurance: detect a group of people who stage accidents to collect on insurance • money laundering: detect suspicious money transactions (US Treasury's Financial Crimes Enforcement Network) • medical insurance: detect professional patients and ring of doctors and ring of references DATA MINING VESIT M.VIJAYALAKSHMI

  11. Other Applications • game statistics to gain competitive advantage Astronomy • JPL and the Palomar Observatory discovered 22 quasars with the help of data mining • IBM Surf-Aid applies data mining algorithms to Web access logs for market-related pages to discover customer preference and behavior pages, analyzing effectiveness of Web marketing, improving Web site organization, etc. DATA MINING VESIT M.VIJAYALAKSHMI

  12. Data Mining Algorithm • Objective: Fit Data to a Model • Descriptive • Predictive • Preference – Technique to choose the best model • Search – Technique to search the data • “Query” DATA MINING VESIT M.VIJAYALAKSHMI

  13. Query Well defined SQL Query Poorly defined No precise query language Database Processing vs. Data Mining Processing • Data • Operational data • Data • Not operational data • Output • Fuzzy • Not a subset of database • Output • Precise • Subset of database DATA MINING VESIT M.VIJAYALAKSHMI

  14. Query Examples • Database • Find all credit applicants with last name of Smith. • Identify customers who have purchased more than $10,000 in the last month. • Find all customers who have purchased milk • Data Mining • Find all credit applicants who are poor credit risks. (classification) • Identify customers with similar buying habits. (Clustering) • Find all items which are frequently purchased with milk. (association rules) DATA MINING VESIT M.VIJAYALAKSHMI

  15. Data Mining: On What Kind of Data? • Relational databases • Data warehouses • Transactional databases • Advanced DB and information repositories • Object-oriented and object-relational databases • Spatial databases • Time-series data and temporal data • Text databases and multimedia databases • Heterogeneous and legacy databases • WWW DATA MINING VESIT M.VIJAYALAKSHMI

  16. Data Mining Models And Tasks DATA MINING VESIT M.VIJAYALAKSHMI

  17. Data Mining Tasks • Prediction Methods • Use some variables to predict unknown or future values of other variables. • Description Methods • Find human-interpretable patterns that describe the data. • Concept description: Characterization and discrimination • Generalize, summarize, and contrast data characteristics, e.g., dry vs. wet regions DATA MINING VESIT M.VIJAYALAKSHMI

  18. Basic Data Mining Tasks • Classification & Prediction • maps data into predefined groups or classes • Finds models (functions) that describe and distinguish classes or concepts for future prediction • E.g., classify countries based on climate, or classify cars based on gas mileage • Presentation: decision-tree, classification rule, neural network • Prediction: Predict some unknown or missing numerical values • 3 methods • Supervised learning • Pattern recognition • Prediction DATA MINING VESIT M.VIJAYALAKSHMI

  19. Basic Data Mining Tasks • Regression • is used to map a data item to a real valued prediction variable. • Learning a function that best fits the target data • Clustering • groups similar data together into clusters. • Class label is unknown: Group data to form new classes, e.g., cluster houses to find distribution patterns • Segmentation • Partitioning DATA MINING VESIT M.VIJAYALAKSHMI

  20. Basic Data Mining Tasks • Summarization maps data into subsets with associated simple descriptions. • Characterization • Generalization • Link Analysis uncovers relationships among data. • Affinity Analysis • Association Rules • age(X, “20..29”) ^ income(X, “20..29K”) à buys(X, “PC”) [support = 2%, confidence = 60%] • contains(T, “computer”) à contains(x, “software”) [1%, 75%] • Sequential Analysis determines sequential patterns. DATA MINING VESIT M.VIJAYALAKSHMI

  21. Sequence Discovery • Given is a set of objects, with each object associated with its own timeline of events, find rules that predict strong sequential dependencies among different events. • Rules are formed by first discovering patterns. • Event occurrences in the patterns are governed by timing constraints. • Patterns similar to association rules but the events are related by time DATA MINING VESIT M.VIJAYALAKSHMI

  22. Are All the “Discovered” Patterns Interesting? • A data mining system/query may generate thousands of patterns, not all of them are interesting. • Interestingness measures: A pattern is interesting if it is easily understood by humans, valid on new or test data with some degree of certainty, potentially useful, novel, or validates some hypothesis that a user seeks to confirm • Objective vs. subjective interestingness measures: • Objective: based on statistics and structures of patterns, e.g., support, confidence, etc. • Subjective: based on user’s belief in the data, e.g., unexpectedness, novelty, etc. DATA MINING VESIT M.VIJAYALAKSHMI

  23. Can We Find All and Only Interesting Patterns? • Find all the interesting patterns: Completeness • Association vs. classification vs. clustering • Search for only interesting patterns: Optimization • Approaches • First general all the patterns and then filter out the uninteresting ones. • Generate only the interesting paterns DATA MINING VESIT M.VIJAYALAKSHMI

  24. Data Mining vs. KDD • Knowledge Discovery in Databases (KDD): process of finding useful information and patterns in data. • Data Mining: Use of algorithms to extract the information and patterns derived by the KDD process. DATA MINING VESIT M.VIJAYALAKSHMI

  25. Data Mining and Business Intelligence Increasing potential to support business decisions End User Making Decisions Business Analyst Data Presentation Visualization Techniques Data Mining Data Analyst Information Discovery Data Exploration Statistical Analysis, Querying and Reporting Data Warehouses / Data Marts OLAP, MDA DBA Data Sources Paper, Files, Information Providers, Database Systems, OLTP DATA MINING VESIT M.VIJAYALAKSHMI

  26. Visualization Techniques • Graphical • Geometric • Icon-based • Pixel-based • Hierarchical • Hybrid DATA MINING VESIT M.VIJAYALAKSHMI

  27. Data Mining: Confluence of Multiple Disciplines Database Technology Statistics Data Mining Machine Learning Visualization Information Science Other Disciplines DATA MINING VESIT M.VIJAYALAKSHMI

  28. Data Mining Development • Similarity Measures • Hierarchical Clustering • IR Systems • Imprecise Queries • Textual Data • Web Search Engines • Relational Data Model • SQL • Association Rule Algorithms • Data Warehousing • Scalability Techniques • Bayes Theorem • Regression Analysis • EM Algorithm • K-Means Clustering • Time Series Analysis • Algorithm Design Techniques • Algorithm Analysis • Data Structures • Neural Networks • Decision Tree Algorithms DATA MINING VESIT M.VIJAYALAKSHMI

  29. Human Interaction Overfitting Outliers Interpretation Visualization Large Datasets High Dimensionality Multimedia Data Missing Data Irrelevant Data Noisy Data Changing Data Integration Application Data Mining Issues DATA MINING VESIT M.VIJAYALAKSHMI

  30. Major Issues in Data Mining (1) • Mining methodology and user interaction • Mining different kinds of knowledge in databases • Interactive mining of knowledge at multiple levels of abstraction • Incorporation of background knowledge • Data mining query languages and ad-hoc data mining • Expression and visualization of data mining results • Handling noise and incomplete data • Pattern evaluation: the interestingness problem • Performance and scalability • Efficiency and scalability of data mining algorithms • Parallel, distributed and incremental mining methods DATA MINING VESIT M.VIJAYALAKSHMI

  31. Major Issues in Data Mining (2) • Issues relating to the diversity of data types • Handling relational and complex types of data • Mining information from heterogeneous databases and global information systems (WWW) • Issues related to applications and social impacts • Application of discovered knowledge • Domain-specific data mining tools • Intelligent query answering • Process control and decision making • Integration of the discovered knowledge with existing knowledge: A knowledge fusion problem • Protection of data security, integrity, and privacy DATA MINING VESIT M.VIJAYALAKSHMI

  32. Social Implications of DM • Privacy • Profiling • Unauthorized use DATA MINING VESIT M.VIJAYALAKSHMI

  33. Data Mining Metrics • Usefulness • Return on Investment (ROI) • Accuracy • Space/Time DATA MINING VESIT M.VIJAYALAKSHMI

  34. Related Concepts Outline Goal: Examine some areas which are related to data mining. • Database/OLTP Systems • Fuzzy Sets and Logic • Information Retrieval(Web Search Engines) • Dimensional Modeling • Data Warehousing • OLAP/DSS • Web Search Engined • Statistics • Machine Learning • Pattern Matching DATA MINING VESIT M.VIJAYALAKSHMI

  35. DB & OLTP Systems • Schema • (ID,Name,Address,Salary,JobNo) • Data Model • ER • Relational • Transaction • Query: SELECT Name FROM T WHERE Salary > 100000 DM: Only imprecise queries output is a KDD object, say a rule a cluster or a classification DATA MINING VESIT M.VIJAYALAKSHMI

  36. Fuzzy Sets and Logic • Fuzzy Set: Set membership function is a real valued function with output in the range [0,1]. • f(x): Probability x is in F. • 1-f(x): Probability x is not in F. • EX: • T = {x | x is a person and x is tall} • Let f(x) be the probability that x is tall • Here f is the membership function DM: Prediction and classification are fuzzy. DATA MINING VESIT M.VIJAYALAKSHMI

  37. Information Retrieval • Information Retrieval (IR): retrieving desired information from textual data. • Library Science • Digital Libraries • Web Search Engines • Traditionally keyword based • Sample query: Find all documents about “data mining”. DM: Similarity measures; Mine text/Web data. DATA MINING VESIT M.VIJAYALAKSHMI

  38. Information Retrieval (cont’d) • Similarity: measure of how close a query is to a document. • Documents which are “close enough” are retrieved. • Metrics: • Precision = |Relevant and Retrieved| |Retrieved| • Recall= |Relevant and Retrieved| |Relevant| DATA MINING VESIT M.VIJAYALAKSHMI

  39. IR Query Result Measures and Classification IR Classification DATA MINING VESIT M.VIJAYALAKSHMI

  40. Dimensional Modeling • View data in a hierarchical manner more as business executives might • Useful in decision support systems and mining • Dimension: collection of logically related attributes; axis for modeling data. • Facts: data stored • Ex: Dimensions – products, locations, date Facts – quantity, unit price DM: May view data as dimensional. DATA MINING VESIT M.VIJAYALAKSHMI

  41. Relational View of Data DATA MINING VESIT M.VIJAYALAKSHMI

  42. Dimensional Modeling Queries • Roll Up: more general dimension • Drill Down: more specific dimension • Dimension (Aggregation) Hierarchy • SQL uses aggregation • Decision Support Systems (DSS): Computer systems and tools to assist managers in making decisions and solving problems. DATA MINING VESIT M.VIJAYALAKSHMI

  43. Data Warehousing • Operational Data: Data used in day to day needs of company. • Informational Data: Supports other functions such as planning and forecasting. • Data mining tools often access data warehouses rather than operational data. DM: May access data in warehouse & couls use OLAP queries DATA MINING VESIT M.VIJAYALAKSHMI

  44. Web Search Engines • Web Search Engines are similar to IR systems • Conventional Search Engines suffer from several problems • Abundance • Limited Coverage • Limited Query • Limited Customization • Concept of “Web Mining” DATA MINING VESIT M.VIJAYALAKSHMI

  45. Statistics • Simple descriptive models • Statistical inference: generalizing a model created from a sample of the data to the entire dataset. • Exploratory Data Analysis: • Data can actually drive the creation of the model • Opposite of traditional statistical view. • Data mining targeted to business user DM: Many data mining methods come from statistical techniques. DATA MINING VESIT M.VIJAYALAKSHMI

  46. Machine Learning • Machine Learning: area of AI that examines how to write programs that can learn. • Often used in classification and prediction • Supervised Learning: learns by example. • Unsupervised Learning: learns without knowledge of correct answers. • Machine learning often deals with small static datasets. DM: Uses many machine learning techniques. DATA MINING VESIT M.VIJAYALAKSHMI

  47. Pattern Matching (Recognition) • Pattern Matching: finds occurrences of a predefined pattern in the data. • Applications include speech recognition, information retrieval, time series analysis. DM: Type of classification. DATA MINING VESIT M.VIJAYALAKSHMI

  48. Data Mining Techniques Outline • Statistical • Point Estimation • Models Based on Summarization • Bayes Theorem • Hypothesis Testing • Regression and Correlation • Similarity Measures • Decision Trees • Neural Networks • Activation Functions • Genetic Algorithms DATA MINING VESIT M.VIJAYALAKSHMI

  49. Similarity Measures • Determine similarity between two objects. • Similarity characteristics: • Alternatively, distance measure measure how unlike or dissimilar objects are. DATA MINING VESIT M.VIJAYALAKSHMI

  50. Distance Measures • Measure dissimilarity between objects DATA MINING VESIT M.VIJAYALAKSHMI

More Related