1 / 27

Another Look at Data Mining

Another Look at Data Mining. Why do we mine? What do we mine? How do we mine?. What is Data Mining. Data mining discovers meaningful new correlations, hidden patterns and relationships in your data Conceptual descendent of statistics Combines machine learning,statistics,and databases

Rita
Download Presentation

Another Look at Data Mining

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Another Look at Data Mining Why do we mine? What do we mine? How do we mine?

  2. What is Data Mining • Data mining discovers meaningful new correlations, hidden patterns and relationships in your data • Conceptual descendent of statistics • Combines machine learning,statistics,and databases • Knowledge discovery:process of building and implementing a data mining solution CS753 Dr. Mary Ann Robbert

  3. Data Mining Overview • Knowledge Discovery in Databases, KDD • No one data mining approach • each tool viewed logically as application of client • Can reside on separate machine or in separate process and access data warehouse • RDBMS or proprietary OLAP embed data mining capabilities deeply within engines to improve efficiency and add extensions • Requires a good foundation in terms of a data warehouse CS753 Dr. Mary Ann Robbert

  4. Data Mining Overview (con’t) • Common algorithmic approaches • association, affinity grouping • predicting, sequence-based analysis • clustering • classification • estimation • Steps are:data selection, data transformation,data mining,result interpretation. CS753 Dr. Mary Ann Robbert

  5. Strategic Benefit of Data Mining • Direct Marketing • Trend Analysis • Fraud detection • Forecasting in Financial Markets CS753 Dr. Mary Ann Robbert

  6. Why Data Mining Now? • Economics • Unprecedented affordability of MIPS and MB • Parallel computing • Enormous amounts of data can be processed • Popularity of data warehouses, data marts • Relatively clean data available CS753 Dr. Mary Ann Robbert

  7. Data Mining compared to Traditional Analysis • Traditional Analysis • Did sales of product X increase in Nov.? • Do sales of product X decrease when there is a promotion on product Y? • Data mining is result oriented • What are the factors that determine sales of product X? CS753 Dr. Mary Ann Robbert

  8. Data Mining compared to Traditional Analysis (con’t) • Traditional; analysis is incremental • Does billing level affect turnover? • Does location affect turnover? • Analyst builds model step by step • Data Mining is result oriented • Identify the factors and predict turnover CS753 Dr. Mary Ann Robbert

  9. Steps in Data Mining • Data Manipulation - can be 70-80% of data mining effort • data cleaning • missing values • data derivation • merging data • Defining a study • Supervised-articulating goal, choosing dependent variable or output and specifying data fields • Unsupervised-group similar types of data or identify exceptions CS753 Dr. Mary Ann Robbert

  10. Steps in Data Mining (con’t) • Reading the data and building the model • model summarizes large amounts of data by accumulating indicators (frequencies,weight,conjunctions,differentiation) • Understanding the model • Know the particular model • Prediction • Choose the best outcome based on historical data CS753 Dr. Mary Ann Robbert

  11. Models • Genetic Algorithms • Neural Nets • Agents • Statistics • Visualization CS753 Dr. Mary Ann Robbert

  12. Genetic Algorithms • Artificial intelligence system that mimics the evolutionary, survival-of-the-fittest processes to generate increasingly better solutions to a problem. • Genetic algorithms produce several generations of solutions, choosing the best of the current set for each new generation. • Examples • Generating human faces based on a few known features. • Generating solutions to routing problems. • Generating stock portfolios. CS753 Dr. Mary Ann Robbert

  13. EVOLUTION IN GENETIC ALGORITHMS • SELECTION - or survival of the fittest. The key is to give preference to better outcomes. • CROSSOVER - combining portions of good outcomes in the hope of creating an even better outcome. • MUTATION- randomly trying combinations and evaluating the success (or failure) of the outcome. CS753 Dr. Mary Ann Robbert

  14. Neural Nets • Mathematical Model of the Way a Brain Functions • Machine learning approach by which historical data can be examined for pattern recognition • A neural network simulates the human ability to classify things based on the experience of seeing many examples. • Pros -Numerical Data • Cons - Opaque, Art or Science CS753 Dr. Mary Ann Robbert ://www.attar.com/

  15. Example • Distinguishing different chemical compounds • Detecting anomalies in human tissue that may signify disease • Reading handwriting • Detecting fraud in credit card use CS753 Dr. Mary Ann Robbert

  16. Intelligent Agents • Software entities that carry out some set of operations on behalf of user or program with some degree of autonomy and employ some knowledge or representation of users goals and desires. • Some common characteristics • ability to communicate, cooperate and coordinate with other agents • ability to act autonomously to achieve collective goal of system CS753 Dr. Mary Ann Robbert

  17. Intelligent Agents (con’t) • Tasks • automate repetitive tasks • finding and filtering information • summarizing complex data • Capability to learn and make recommendations • Black box approach hides complexity and allows for design of scalable system CS753 Dr. Mary Ann Robbert

  18. Comparison Starting Information Expert’s know-how Acceptable patterns Set of possible solutions Your preferences AI System Expert Systems Neural Networks Genetic Algorithms Intelligent Agents Problem Type Diagnostic or prescriptive Identification, classification, prediction Optimal solution Specific and repetitive tasks Based On Strategies of experts The human brain Biological evolution One or more AI techniques

  19. Statistics • SAS, SPSS • Pros - Established technology • Cons - Needs assumptions, nominal variable handling, management acceptance? CS753 Dr. Mary Ann Robbert

  20. Visualization • Data visualization refers to technologies that support visualization of information • Includes – digital images, GIS, multi-dimensions, 3-D presentations, animations • http://www.almaden.ibm.com/cs/quest/demo/assoc/general.html CS753 Dr. Mary Ann Robbert

  21. Data Mining is Not a Silver Bullet • It does not: • Find answers to questions you don’t ask • Eliminate the need for domain experience • Remove the need for data analysis skills CS753 Dr. Mary Ann Robbert

  22. Data Mining Software • http://www.kdnuggets.com/software/ • http://www.attar.com/ download • http://www.cs.bham.ac.uk/~anp/software.html software listing CS753 Dr. Mary Ann Robbert

  23. Six Rules of Data Qualityby Ken Orr 1. Data that is not used cannot be correct for very long 2. Data Quality in an information system is a function of its use, not its collection 3.Data quality will ultimately be no better than its most stringent use 4. Data quality problems tend to become worse with the age of the system 5. Less likely it is that some data element will change, more traumatic it will be when it finally does change. 6. Information overload affects data quality CS753 Dr. Mary Ann Robbert

  24. Data Quality Software • http://www.rulequest.com/gritbot-info.html CS753 Dr. Mary Ann Robbert

  25. General DW Data transformation • Resolve inconsistent legacy formats • Strip out unwanted fields • Interpret codes into text • Combine data from multiple sources under a common key • Find fields used for multiple purposes and interpret fields value based on context CS753 Dr. Mary Ann Robbert

  26. Data transformation for Data Mining • Flag normal, abnormal, out of bounds or impossible facts • Recognize random or noise values from context and mask out • Apply uniform treatment to NULL values • Flag fast records with changed status • Classify individual record by one of its aggregates CS753 Dr. Mary Ann Robbert

  27. Conclusion • For successful data mining: • data analysis and mining goals must be identifies and formulated • appropriate data must be selected, cleaned and prepared for queries and business analysis • http://www.rulequest.com/cubist-examples.html#BOSTON • http://www.almaden.ibm.com/cs/quest/ CS753 Dr. Mary Ann Robbert

More Related