Business Intelligence: A Managerial Approach (2nd Edition)

1. Business Intelligence:A Managerial Approach(2nd Edition) Chapter 4: Data Mining for Business Intelligence

2. 4.1 DATA MINING CONCEPTS AND DEFINITIONS • It used to understanding customers, vendors, business processes, and the extended supply chain very well. • Although the term data mining is relatively new, the ideas behind it are not. • Why, then, has it suddenly gained the attention of the business world??

3. 4.1 DATA MINING CONCEPTS AND DEFINITIONS • WHY DATA MINING ? • More intense competition at the global scale • Recognition of the value in data sources • Availability of quality data on customers, vendors, transactions, Web, etc. • Consolidation and integration of data repositories into data warehouses • The exponential increase in data processing and storage capabilities; and decrease in cost • Movement toward conversion of information resources into nonphysical form

4. Definitions, Characteristics, and Benefits • Definitions: • is a term used to describe discovering or "mining" knowledge from large amounts of data. “knowledge mining” • Technically speaking, data mining is a process that uses statistical, mathematical, and artificial intelligence techniques to extract and identify useful information and subsequent knowledge (or patterns) from large sets of data.

5. Definition • The nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data stored in structured databases • Keywords in this definition: Process, nontrivial , valid, novel, potentially useful, understandable • Other names: knowledge extraction, pattern analysis, knowledge discovery, information harvesting, pattern searching, data dredging

6. Data Mining Characteristics/Objectives • Source of data for DM is often a consolidated data warehouse (not always!). • DM environment is usually a client-server • Sophisticated new tools, including advanced visualization tools, help to remove the information ore buried in corporate files or archival public records • The miner is often an end user. • Striking it rich requires creative thinking. • Data mining tools are readily combined with spreadsheets and other software development tools. • Because of the large amounts of data and massive search efforts, it is sometimes necessary to use parallel processing for data mining.

7. Data Mining at the Intersection of Many Disciplines

8. What Does DM Do?How Does it Work? • DM extracts patterns from data •  Pattern? A mathematical (numeric and/or symbolic) relationship among data items • Types of patterns •  Association •  Prediction •  Cluster (segmentation) •  Sequential (or time series) relationships

9. A Taxonomy for Data Mining Tasks.

10. Data Mining Tasks. • PREDICTION :is commonly referred to as the act of telling about the future. • prediction can be named more specifically as classification: (where the predicted thing, such as tomorrow's forecast, is a class label such as "rainy" or "sunny") or regression:(where the predicted thing, such tomorrow's temperature, is a real number such as "65°F"). • Classification, is perhaps the most common of all data mining tasks. The objective of classification is to analyze the historical data stored in a database and automatically generate a model that can predict future behavior. • (Neural networks OR Decision trees )

11. Data Mining Tasks. • Clustering: partitions a collection of things (e.g. , objects and events presented in a structured dataset) into segments (or natural groupings) whose members share similar characteristics. • market segmentation with cluster analysis. OR segmenting customers.

12. Data Mining Tasks. • Associations: is a popular and well-researched technique for discovering interesting relationships among variables in large databases • In the context of the retail industry , association rule mining is often called market-basket analysis

13. Data Mining Applications • Customer Relationship Management • Banking & Other Financial • Retailing and Logistics • Manufacturing and Maintenance • Brokerage and Securities Trading • Insurance • Computer hardware and software • Science and engineering • Government and defense • Homeland security and law enforcement • Travel industry • Healthcare • Medicine • Entertainment industry • Sports • Etc

14. Data Mining Process: CRISP-DM Cross-Indust1y Standard Process proposed in the mid-1990s by a European consortium of companies

15. Data Mining Process: CRISP-DM Step 1: Business Understanding know what the study is for like ("What are the common characteristics of the customers we have) Step 2: Data Understanding identify the relevant data from many available databases. quantitative OR qualitative Step 3: Data Preparation (!) (table 4.4) • called as data preprocessing .take the data identified in the previous step and prepare them for analysis by data mining • data preprocessing consumes the most time and effort; most believe that this step accounts for roughly 80% of the total time.

16. Data Mining Process: CRISP-DM Step 4: Model Building use a variety of data mining methods and algorithms Step 5: Testing and Evaluation a critical and challenging task Step 6: Deployment The deployment step may also include maintenance activities

17. Data Preparation – A Critical DM Task (see table 4.4)

18. Data Mining Process: SEMMA

19. Data Mining Process Source: KDNuggets.com, August 2007

20. Data Mining Methods: Classification • Most frequently used DM method • Part of the machine-learning family • Employ supervised learning • Learn from past data, classify new data • The output variable is categorical (nominal or ordinal) in nature

21. Classification Techniques • Decision tree analysis • Statistical analysis • Neural networks • Support vector machines • Case-based reasoning • Bayesian classifiers • Genetic algorithms • Rough sets

22. Decision Trees • Employs the divide and conquer method • Recursively divides a training set until each division consists of examples from one class • Create a root node and assign all of the training data to it • Select the best splitting attribute • Add a branch to the root node for each value of the split. Split the data into mutually exclusive subsets along the lines of the specific split • Repeat the steps 2 and 3 for each and every leaf node until the stopping criteria is reached A general algorithm for decision tree building

23. Decision Tree

24. Cluster Analysis for Data Mining • Used for automatic identification of natural groupings of things • Part of the machine-learning family • Employ unsupervised learning • Learns the clusters of things from past data, then assigns new instances • There is not an output variable • Also known as segmentation

25. Cluster Analysis for Data Mining • Clustering results may be used to • Identify natural groupings of customers • Identify rules for assigning new cases to classes for targeting/diagnostic purposes • Provide characterization, definition, labeling of populations • Decrease the size and complexity of problems for other data mining methods • Identify outliers in a specific domain (e.g., rare-event detection)

26. Cluster Analysis for Data Mining • Analysis methods • Statistical methods (including both hierarchical and nonhierarchical), such as k-means, k-modes, and so on • Neural networks (adaptive resonance theory [ART], self-organizing map [SOM]) • Fuzzy logic (e.g., fuzzy c-means algorithm) • Genetic algorithms • Divisive versus Agglomerative methods

27. Cluster Analysis for Data Mining • k-Means Clustering Algorithm • k : pre-determined number of clusters • Algorithm (Step 0: determine value of k) Step 1: Randomly generate k random points as initial cluster centers Step 2: Assign each point to the nearest cluster center Step 3: Re-compute the new cluster centers Repetition step: Repeat steps 3 and 4 until some convergence criterion is met (usually that the assignment of points to clusters becomes stable)

28. Cluster Analysis for Data Mining- k-Means Clustering Algorithm

29. Association Rule Mining • A very popular DM method in business • Finds interesting relationships (affinities) between variables (items or events) • Part of machine learning family • Employs unsupervised learning • There is no output variable • Also known as market basket analysis • Often used as an example to describe DM to ordinary people, such as the famous “relationship between diapers and beers!”

30. Association Rule Mining • Input: the simple point-of-sale transaction data • Output: Most frequent affinities among items • Example: according to the transaction data… “Customer who bought a laptop computer and a virus protection software, also bought extended service plan 70 percent of the time." • How do you use such a pattern/knowledge? • Put the items next to each other for ease of finding • Promote the items as a package (do not put one on sale if the other(s) are on sale) • Place items far apart from each other so that the customer has to walk the aisles to search for it, and by doing so potentially seeing and buying other items

31. Association Rule Mining • A representative applications of association rule mining include • In business: cross-marketing, cross-selling, store design, catalog design, e-commerce site design, optimization of online advertising, product pricing, and sales/promotion configuration • In medicine: relationships between symptoms and illnesses; diagnosis and patient characteristics and treatments (to be used in medical DSS); and genes and their functions (to be used in genomics projects)…

32. Association Rule Mining • Are all association rules interesting and useful? A Generic Rule: X  Y [S%, C%] X, Y: products and/or services X: Left-hand-side (LHS) Y: Right-hand-side (RHS) S:Support: how often X and Y go together C:Confidence: how often Y go together with the X Example: {Laptop Computer, Antivirus Software}  {Extended Service Plan} [30%, 70%]

33. Association Rule Mining • Apriori Algorithm • Finds subsets that are common to at least a minimum number of the itemsets • uses a bottom-up approach • frequent subsets are extended one item at a time (the size of frequent subsets increases from one-item subsets to two-item subsets, then three-item subsets, and so on), and • groups of candidates at each level are tested against the data for minimum support • see the figure…

34. Association Rule Mining • Apriori Algorithm

35. Problem Decomposition – Example X  Y Support = X and Y / T Confidence = supp ( x,y) / supp ( X) For the rule Shoes  Jacket • Support = Sup({Shoes,Jacket)}= 2/4= .5 =50% • Confidence =0.5 / 0.75 =66.6% Jacket Shoes has 50% support and 66.6% confidence

38. Element of ANN • PROCESSING ELEMENTS (PE) The PE of an ANN are essentially artificial neurons. Similar to biological neurons. • INFORMATION PROCESSING The inputs received by a neuron go through a two-step process to turn into outputs: summation function and transformation function • NETWORK STRUCTURE Each ANN is composed of a collection of neurons (or PE) that are grouped into layers

39. Text and Web Data mining

40. Text Mining Concepts • 85-90 percent of all corporate data is in some kind of unstructured form (e.g., text) • Unstructured corporate data is doubling in size every 18 months • Tapping into these information sources is not an option, but a need to stay competitive • Answer: text mining • A semi-automated process of extracting knowledge from unstructured data sources • a.k.a. text data mining or knowledge discovery in textual databases

41. Data Mining versus Text Mining • Both seek for novel and useful patterns • Both are semi-automated processes • Difference is the nature of the data: • Structured versus unstructured data • Structured data: in databases • Unstructured data: Word documents, PDF files, text excerpts, XML files, and so on • Text mining – first, impose structure to the data, then mine the structured data

42. Text Mining Concepts • Benefits of text mining are obvious especially in text-rich data environments • e.g., law (court orders), academic research (research articles), finance (quarterly reports), medicine (discharge summaries), biology (molecular interactions), technology (patent files), marketing (customer comments), etc. • Electronic communization records (e.g., Email) • Spam filtering • Email prioritization and categorization • Automatic response generation

43. Text Mining Application Area • Information extraction • Topic tracking • Summarization • Categorization • Clustering • Concept linking • Question answering

44. Web Mining Overview • Web is the largest repository of data • Data is in HTML, XML, text format • Challenges (of processing Web data) • The Web is too big for effective data mining • The Web is too complex • The Web is too dynamic • The Web is not specific to a domain • The Web has everything • Opportunities and challenges are great!

45. Web Mining • Web mining (or Web data mining) is the process of discovering intrinsic relationships from Web data (textual, linkage, or usage)