Data Mining UMUC CSMN 667

1. Data Mining UMUC CSMN 667 Lecture #1 UMUC Data Mining Lecture 1

2. So what is it? Data Mining is “an information extraction activity whose goal is to discover hidden facts contained in large databases.” UMUC Data Mining Lecture 1

3. Class Textbooks • Margaret Dunham’s book: “Data Mining Introductory and Advanced Topics” • from Prentice Hall • numerous publication dates listed (2002/2003) • there is only one edition (just buy it) • APA Style Guide: Publication manual of the American Psychological Association (2001, 5th ed.) - required by UMUC UMUC Data Mining Lecture 1

4. Additional Assignment for first month • Set up database account on our class database server: dbcourse3.umuc.edu • Refer to WebTycho for instructions: • Change your passwords immediately in 2 places: your Unix server account and your Oracle database account (both passwords are initially the same, but they are completely independent). UMUC Data Mining Lecture 1

5. Reminders • The word “DATA” is plural. The singular form of the word is “datum” -- one datum is okay, but many data are better. • Time is what prevents everything from happening at once. So, please use good time management skills to keep from falling behind in your reading and other class assignments. UMUC Data Mining Lecture 1

6. “Data Mining 101” An Introduction to Data Mining Data mining is defined as “an information extraction activity whose goal is to discover hidden facts contained in (large) databases." UMUC Data Mining Lecture 1

7. Evolution of Data Mining<http://www.thearling.com/text/dmwhite/dmwhite.htm> UMUC Data Mining Lecture 1

8. Data Mining is Ready for Prime Time • Data mining is ready for general application because it engages three technologies that are now sufficiently mature: • Massive data collection & delivery • Powerful multiprocessor computers • Sophisticated data mining algorithms UMUC Data Mining Lecture 1

9. 6 Business Reasons to use Data Mining • Most organizations already collect and refine massive quantities of data. • Their most important information is in their data warehouses. • Data mining moves beyond the analysis of past events … to predicting future trends and behaviors that may be missed because they lie outside the experts’ expectations. • Data mining tools can answer complex business questions that traditionally were too time-consuming to resolve. • Data mining tools can explore the intricate interdependencies within databases in order to discover hidden patterns and relationships. • Data mining allows decision-makers to make proactive, knowledge-driven decisions. UMUC Data Mining Lecture 1

10. Another Business Reason to use Data Mining UMUC Data Mining Lecture 1

11. A Key Concept for Data Mining • Data Mining delivers actionable data: • data that support decision-making • data that lead to knowledge and understanding • data with a purpose • i.e., Data do not exist for their own sake. • The Data Warehouse is a corporate asset (whether in business, marketing, banking, science, telecommunications, entertainment, computer security, or Homeland Security). UMUC Data Mining Lecture 1

12. Data Mining - the up side • Data mining is everywhere: • Huge scientific databases (NASA, Human Genome,…) • Corporate databases (OLAP) • Credit card usage histories (Capital One) • Loan applications (Credit Scoring) • Customer purchase records (CRM) • Web traffic analysis (Doubleclick) • Network security intrusion detection (Silent Runner) • The hunt for terrorists (DARPA TIA) • The NBA! … the NBA?? UMUC Data Mining Lecture 1

13. Data Mining - the down side • Data mining is a pejorative in the business database community (“data dredging”) • They prefer to call it Knowledge Discovery, or Business Intelligence, or CRM (Customer Relationship Management), or Marketing, or OLAP (On-Line Analytical Processing) • The Data Mining Moratorium Act of 2003 • see first page of the bill on next slide • debated within the U.S.Congress • privacy concerns • directly primarily against the DARPA TIA Program (Total Information Awareness) UMUC Data Mining Lecture 1

14. http://www.cdt.org/legislation/108th/privacy/030122feingold.pdfhttp://www.cdt.org/legislation/108th/privacy/030122feingold.pdf 108TH CONGRESS 1ST SESSIONS.__________ IN THE SENATE OF THE UNITED STATES Mr. FEINGOLD introduced the following bill; which was read twice and referred to the Committee on _________________ A BILL To impose a moratorium on the implementation of datamining under the Total Information Awareness program of the Department of Defense and any similar program of the Department of Homeland Security, and for other purposes. 1 Be it enacted by the Senate and House of Representa- 2 tives of the United States of America in Congress assembled, 3 SECTION 1. SHORT TITLE. 4This Act may be cited as the ‘‘Data-Mining Morato- 5 rium Act of 2003’’. 6 SEC. 2. FINDINGS. 7Congress makes the following findings: UMUC Data Mining Lecture 1

15. The Information Age is Here! • "Data doubles about every year, but useful information seems to be decreasing." • Margaret Dunham, "Data Mining Techniques & Algorithms", 2002 • "There is a growing gap between the generation of data and our understanding of it." • Witten & Frank, "Data Mining: Practical Machine Learning Tools", 1999 • "The trouble with facts is that there are so many of them" • Samuel McChord Crothers, "The Gentle Reader", 1973 • "Get your facts first, and then you can distort them as much as you please." • Mark Twain UMUC Data Mining Lecture 1

16. Characteristics of The Information Age: • Data “Avalanche” • the flood of Terabytes of data is already happening, whether we like it or not • our present techniques of handling these data do not scale well with data volume • Distributed Digital Archives • will be the main access to data • will need to handle hundreds to thousands of queries per day • Systematic Data Exploration and Data Mining • will have a central role • statistical analysis of “typical” events • automated search for “rare” events UMUC Data Mining Lecture 1

17. The Data Flood is Everywhere • Huge quantities of data are being generated in all business, government, and research domains: • Banking, retail, marketing, telecommunications, other business transactions ... • Scientific data: genomics, astronomy, biology, etc. • Web, text, and e-commerce UMUC Data Mining Lecture 1

18. 5 million terabytes created in 2002 UC Berkeley 2003 estimate: 5 exabytes (5 million terabytes) of new data were created in 2002. http://www.sims.berkeley.edu/research/projects/how-much-info-2003/ What is a gigabyte, terabyte, petabyte, exabyte, …? Look at the definitions and examples in the following article: http://www.jamesshuggins.com/h/tek1/how_big.htm UMUC Data Mining Lecture 1

19. Data Growth Rate • Twice as much information was created in 2002 as in 1999 (~30% annual growth rate). • Other growth rate estimates are even higher. • Very little of these data will ever be looked at by a human. • Data Mining is NEEDED to make sense of and to make use of these data. UMUC Data Mining Lecture 1

20. What is Data Mining? • Data mining is defined as “an information extraction activity whose goal is to discover hidden facts contained in (large) databases." • Data mining is used to find patterns and relationships in data. (EDA = Exploratory Data Analysis) • Patterns can be analyzed via 2 types of models: • Descriptive : Describe patterns and create meaningful subgroups or clusters. • Predictive : Forecast explicit values, based upon patterns in known results. • How does this become useful (not just bits of data)? ... • … through KNOWLEDGE DISCOVERY Data  Information  Knowledge  Understanding / Wisdom! UMUC Data Mining Lecture 1

21. Historical Note: Many Names of Data Mining • Data Fishing, Data Dredging: 1960- • used by Statisticians (as a bad name) • Data Mining :1990- • used by DB & business communities • in 2003 – bad image because of DARPA TIA • Knowledge Discovery in Databases (1989-) • used by AI & Machine Learning communities • also Data Archaeology, Information Harvesting, Information Discovery, Knowledge Extraction, ... Currently: Data Mining and Knowledge Discovery are used interchangeably. UMUC Data Mining Lecture 1

22. Data Mining Examples • Classic Textbook Example of Data Mining(Legend?): Data mining of grocery store logs indicated that men who buy diapers also tend to buy beer at the same time. • Blockbuster Entertainment mines its video rental history database to recommend rentals to individual customers. • A financial institution discovered that credit applicants who used pencil on the form were much more likely to default on their debts than those who filled out the application using ink. • Credit card companies recommend products to cardholders based on analysis of their monthly expenditures. • Airline purchase transaction logs revealed that 9-11 hijackers bought one-way airline tickets with the same credit card. • Astronomers examined objects with extreme colors in a huge database to discover the most distant Quasars ever seen. UMUC Data Mining Lecture 1

23. UMUC Data Mining Lecture 1

24. Data Mining Application:Marketing • Sales Analysis • associations between product sales: • beer and diapers • strawberry pop tarts and beer (and hurricanes) • Customer Profiling • data mining can tell you what types of customers buy what products • Identifying Customer Requirements • identify the best products for different customers • use prediction to find what factors will attract new customers UMUC Data Mining Lecture 1

25. Data Mining Application:Fraud Detection • Auto Insurance Fraud • Association Rule Mining can detect a group of people who stage accidents to collect on insurance • Money Laundering • Since 1993, the US Treasury's Financial Crimes Enforcement Network agency has used a data-mining application to detect suspicious money transactions • Banking: Loan Fraud • Security Pacific/Bank of America uses data mining to help with commercial lending decisions and to prevent fraud UMUC Data Mining Lecture 1

26. The Necessity of Data Mining • Enormous interest in these data collections. • The environment to exploit these data does not exist! • 1 Terabyte at 100 Mbits/sec takes 1 day to transfer. • Hundreds to thousands of queries per day. • Data will reside at multiple locations, in many different formats. • Existing analysis tools do not scale to Terabyte data collections. • The need is acute! A solution will not just happen. UMUC Data Mining Lecture 1

27. What is Knowledge Discovery? • Knowledge discovery refers to “finding out new knowledge about an application domain using data on the domain usually stored in a database.” • Application domains: scientific, customer purchase records, computer network logs, web traffic logs, financial transactions, census data, basketball play-by-play histories, ... • Why are Data Mining & Knowledge Discovery such hot topics? --- because of the enormous interest in these huge databases and their potential for new discoveries. • In large databases, Data Mining and Knowledge Discovery come in two flavors: • Event-based mining • Relationship-based mining UMUC Data Mining Lecture 1

28. Event-Based Mining (Event-based mining is based upon events or trends in data.) Four distinct orthogonal categorizations: • Known events / known models - use existing models (descriptive models) to locate known phenomena of interest either spatially or temporally within a large database. • Known events / unknown models - use clustering properties of data to discover new relationships and patterns among known phenomena. • Unknown events / known models - use known associations and relationships (predictive models) among parameters that describe a phenomenon to predict the presence of previously unseen examples of the same phenomenon within a large complex database. • Unknown events / unknown models - use thresholds or trends to identify transient or otherwise unique ("one-of-a-kind") events and therefore to discover new phenomena.  Serendipity! UMUC Data Mining Lecture 1

29. Relationship-Based Data Mining(Based upon associations & relationships among data items) • Spatial associations -- identify events or objects at the same physical spatial location, or at related locations (e.g., urban versus rural data). • Temporal associations -- identify events or transactions occurring during the same or related periods of time (e.g., periodically, or N days after event X). • Coincidence associations -- use clustering techniques to identify events that are co-located (that coincide) within a multi-dimensional parameter space. UMUC Data Mining Lecture 1

30. Event-Based Mining (EBM) - Homeland Security Example • Known events / known models - use existing models (descriptive models) to locate known phenomena of interest within a large database. • e.g., Terrorist activities have been financed through certain organizations. Search for similar transactions in large financial databases. • Known events / unknown models - use clustering properties of data to discover new relationships and patterns among known phenomena. • e.g., Search through credit card, travel, and phone histories of 9-11 hijackers to discover previously unknown characteristics and behavior patterns of terrorists. • Unknown events / known models - use known associations and relationships (predictive models) among parameters that describe a phenomenon to predict the presence of previously unseen examples within a large complex database. • e.g., Use knowledge of terrorist behavior patterns (e.g, heightened phone activity) to identify new terrorists and/or to raise new terrorist alerts. • Unknown events / unknown models - use thresholds or trends or outlier detection to identify transient or otherwise unique ("one-of-a-kind") events, and therefore to discover new phenomena. • e.g., Explore all known data (including intelligence, news reports, e-mail, credit card histories, phone records, organizational memberships) to identify new threats. (EBM is based upon events or trends in data.) UMUC Data Mining Lecture 1

31. Relationship-Based Mining (RBM) -Homeland Security Example • Spatial associations -- identify events (e.g,airline ticket purchases) occurring at the same location in some geospatial parameter space (e.g,travel on the same flights). • Temporal associations -- identify events occurring during the same or related periods of time (e.g, airline ticket purchasesfor travel on the same flightspurchased at the same time). • Coincidence associations -- use clustering techniques to identify events that are co-located within a multi-dimensional parameter space (e.g, airline ticketsfor the same flightspurchased at the same time as one-way tickets on the same credit card, with travelers of Mid-East origin, having recent U.S. entry, were students in flight schools, having records of numerous phone calls to Afghanistan, and having visited Hamburg Germany at some time in the past few years). (RBM is based upon associations and relationships among data items.) UMUC Data Mining Lecture 1

32. User Requirements for a Data Mining System(What features must a D.M. system have for your users?) • Cross-Identification - refers to the classical problem of associating the objects listed in one database to the objects listed in another. • Cross-Correlation - refers to the search for correlations, tendencies, and trends between parameters in multi-dimensional data, usually across databases. • Nearest-Neighbor Identification - refers to the general application of clustering algorithms in multi-dimensional parameter space, usually within a single database. • Systematic Data Exploration - refers to the application of the broad range of event-based and relationship-based queries to one or more databases in the hope of making a serendipitous discovery of new events/objects or a new class of events/objects. UMUC Data Mining Lecture 1

33. Representative Data Mining Architecture<http://www.thearling.com/text/dmwhite/dmwhite.htm> UMUC Data Mining Lecture 1

34. Data leads to Knowledge leads to Understanding Data  Information  Knowledge  Understanding / Wisdom! Remember what we said earlier : EXAMPLE : • Data = 00100100111010100111100 (stored in database) • Information = ages and heights of children (metadata) • Knowledge = the older children tend to be taller • Understanding = children’s bones grow as they get older UMUC Data Mining Lecture 1

35. Astronomy Example Data: (a) Imaging data (ones & zeroes) (b) Spectral data (ones & zeroes) Information (catalogs / databases): • Measure brightness of galaxies from image (e.g., 14.2 or 21.7) • Measure redshift of galaxies from spectrum (e.g., 0.0167 or 0.346) Knowledge: Hubble Diagram  Redshift-Brightness Correlation  Redshift = Distance Understanding: the Universe is expanding!! UMUC Data Mining Lecture 1

36. Goal of Data Mining • The end goal of data mining is not the data themselves, but the new knowledge and understanding that are revealed in the process = Business Intelligence (BI). (Remember what we said about the business community’s opinion of D.M.) • This is why the research field is usually referred to asKDD = Knowledge Discovery in Databases. UMUC Data Mining Lecture 1

37. Some words of wisdom • "We have confused information (of which there is too much) with ideas (of which there are too few)." • Paul Theroux • "The great Information Age is really an explosion of non-information; it is an explosion of data ... it is imperative to distinguish between the two; information is that which leads to understanding." • R.S. Wurman in his book: Information Anxiety2 UMUC Data Mining Lecture 1

38. The Data Mining Process(more about this later) The most important and time-consuming step is Cleaning the Data. UMUC Data Mining Lecture 1

39. Data Mining Methods and Some Examples Clustering Classification Associations Neural Nets Decision Trees Pattern Recognition Correlation/Trend Analysis Principal Component Analysis Regression Analysis Outlier/Glitch Identification Visualization Autonomous Agents Self-Organizing Maps (SOM) Link (Affinity) Analysis Find all groups and classes of objects represented in the data Classify new data items using the known classes & groups Find associations and patterns among different data items Organize information in the database based on relationships among key data descriptors Identify linkages between data items based on featuresshared in common UMUC Data Mining Lecture 1

40. Some Data Mining Techniques Graphically Represented Self-Organizing Map (SOM) Clustering Neural Network Link Analysis Decision Tree Outlier (Anomaly) Dectection UMUC Data Mining Lecture 1

41. Remember what it is … Data Mining is “an information extraction activity whose goal is to discover hidden facts contained in large databases.” UMUC Data Mining Lecture 1

42. Data Mining Technique: Clustering In this case, three different groups (classes) of items were found among all of the items in the data set. UMUC Data Mining Lecture 1

43. Data Mining Technique: Decision Tree Classification Question: Should I play tennis today? Similar to game “20 questions” Same technique used by bank loan officers to identify good potential customers versus poor customers. (I must really love tennis!) UMUC Data Mining Lecture 1

44. Data Mining Technique:Association Rule Mining(Market Basket Analysis) transaction id products bought customer id sales records: • Trend (Rule): Products p5, p8 often bought together • Trend (Rule): Customer 12 likes product p9 UMUC Data Mining Lecture 1

45. Data Mining Algorithm: The SOM Figure: The SOM (Self-Organizing Map) is one technique for organizing information in a database based upon links between concepts. It can be used to find hidden relationships and patterns in more complex data collections, usually based on links between keywords or metadata. UMUC Data Mining Lecture 1

46. Data Mining Application: Outlier Detection Figure: The clustering of data clouds (dc#) within a multidimensional parameter space (p#). Such a mapping can be used to search for and identify clusters, voids, outliers, one-of-kinds, relationships, and associations among arbitrary parameters in a database (or among various parameters in geographically distributed databases). UMUC Data Mining Lecture 1

47. Link Analysis for Homeland Security: Find all connections and relationships among known terrorists. UMUC Data Mining Lecture 1

48. Data Mining Technology:Parallel Mining Figure: Parallel Data Mining The application of parallel computing resources and parallel data access (e.g., RAID) enables concurrent drill-downs into large data collections UMUC Data Mining Lecture 1

49. Data Mining Methods Explained • Clustering: Group data items according to tight relationships. • Classification: Assign data items to predetermined groups. • Associations: Associate data with similar relationships. The beer-diaper example is an example of associative mining. • Artificial Neural Networks (ANN): Non-linear predictive models that learn through training and resemble biological neural networks in structure. • Decision Trees: Hierarchical sets of decisions, based upon rules, for rapid classification of a data collection. • Sequential Patterns: Identify or predict behavior patterns or trends. • Genetic Algorithms: Rapid optimization techniques that are based on the concepts of natural evolution. • Nearest Neighbor Method: Classify a data item according to its nearest neighbors (records that are most similar). • Rule induction: The extraction of useful if-then rules from data based on statistical significance. • Data visualization: The illustration and visual interpretation of complex relationships in multidimensional data using graphics tools. • Self-Organizing Map (SOM): Graphically organizes (in a 2-dimensional map) the information stored within a database based upon similarities and links between concepts. It can be used to find hidden relationships and patterns in more complex data collections. UMUC Data Mining Lecture 1

50. Data Mining Techniques: techniques are based on Algorithms; techniques are used in Applications UMUC Data Mining Lecture 1