1 / 39

Data Mining

Data Mining. Brad Morantz PhD. What is a Mountain?. A big pile of dirt!. Data ?. Some people would say that data on them is . . . Dirt A big pile of data is a . . . Mountain of data or a data warehouse. Mountain of data. Bob has blue Ford. Chen has $10. Mary eats rice krisp.

yered
Download Presentation

Data Mining

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Data Mining Brad Morantz PhD

  2. Brad Morantz What is a Mountain? • A big pile of dirt!

  3. Brad Morantz Data ? • Some people would say that data on them is . . . • Dirt • A big pile of data is a . . . • Mountain of data or a data warehouse

  4. Brad Morantz Mountain of data Bob has blue Ford Chen has $10 Mary eats rice krisp

  5. Brad Morantz Notice in Last Example . . • Different people • Different facts • Unconnected? • Maybe: • Bob bought the Ford from Chen (for $10?) • Mary bought the Rice Krisp from Chen • Maybe Bob delivers the Rice Krisp in his Ford • Maybe something else • Maybe nothing

  6. Brad Morantz Data Mining • Is the nontrivial extraction of implicit previously unknown and potentially useful information and data. (James Buckley) • Machine Learning employs search heuristics to uncover interesting and systematic relationships in data. (Dhar & Stein)

  7. Brad Morantz Data Mining • Going into a data file and looking for patterns and relationships • Use of historical data to find patterns and improve future decisions • Also called KDD • Knowledge discovery from databases

  8. Brad Morantz Uses • Merchandising • Association rules • Customer profiling • Problem solving & risk profiling • Medical • Financial • Pattern Recognition • IRS • NSA • Many more

  9. Brad Morantz We want information that is: • Original • Nontrivial • Fundamental • Simple • Useful • Currently needs to be numerical or categorical

  10. Brad Morantz Theoretical Definitionof Data Mining • Go into data • No preconceived ideas • See what is found

  11. Brad Morantz Analogy • Somebody gives you a box of stuff • You have no idea what is in it • You open it up • You look through the contents • You make note of all that you have found • You may sort the stuff into piles based upon some similarities

  12. Brad Morantz Association Rule Example • Check out at store • Each check out is a record • Called a “basket” • Look at what was purchased • Look for relationships or associated items • milk and cereal, beer and pizza

  13. Brad Morantz Grocery Example Invoice 1 2 3 4 5 6 Milk X X X X Cereal X X X Pizza X X X Beer X X X Eggs X X X X

  14. Brad Morantz Milk AND Cereal • Support • Percent of baskets where rule is true • P(milk AND cereal) = 3/6 = 0.500 • Confidence • Percent of baskets that have cereal given milk is in the basket • P(cereal|milk) = ¾ = 0.75

  15. Brad Morantz Benefits to Retailer • Store layout & increased sales • Puts items that sell together close to each other • Increase impulse purchases • Help forecast inventory levels • Learn customer preferences • Increase profitability

  16. Brad Morantz Common Meaningof Data Mining • Go into data • Look for patterns • Look for answers • Look for similarities • Look for rules • Look for relationships

  17. Brad Morantz More Analogy • Someone gives you the box full of stuff • You are looking for hit singles by Mick Jagger (very specific) • Along the way, you might discover that they are round, plastic, about 7” diameter, with a 1” hole in the middle • You might discover that they are on London label • You might discard all other things in the box

  18. Brad Morantz Example • High risk pregnancy • Tons of data • What are common factors? • Myocardial Infarction (MCI) • Tons of data • What factors common in non-occurrence? • Loan application • Many other situations

  19. Brad Morantz Simple Use of Common Factors • Nurse/banker/etc can ask a few questions • The key factors • Identified by data mining • Now has indication of risk or outcome

  20. Brad Morantz Interesting Finding • Data mining stolen credit cards • Common pattern • Pay at pump authorization • To determine if card is good • Fast indication of stolen card

  21. Brad Morantz What is in the Mountain of data? • Better yet, ask • What is NOT in the mountain of data?

  22. Brad Morantz Original • US Navy • Boilers on boats exploding

  23. Brad Morantz Think about a trip to the store • They scan your purchases • You give them credit card or check • They have your • name • Address • Telephone number • Maybe drivers license • Maybe SSN • List of items that you bought, including size

  24. Brad Morantz Mother with kids Milk Bread Eggs Cereal If diapers, then infants Grocery Store • Single male • Pizza • Beer • Potato chips

  25. Brad Morantz Think About . . • All of the “baskets” about you out there • Medical Information Bureau (MIB) • Charge Cards • Bank information • Public records • Store purchases • Travel records • Much more • The combination of all this creates a pattern that describes you exactly

  26. Brad Morantz Applications • Marketing • Target marketing • Store layout • Better understanding • Medical • Criminal • Financial • Science

  27. Brad Morantz Methods • Rule Induction • Similarity Engine • OLS & logistic regression • Data visualization • Bayesian Classification & networks • Clustering • Specialized programs • Clementine • Enterprise Miner

  28. Brad Morantz Plot • Plot data and put in breaks to split it Months on the job Bad credit risks Good credit risks Debt ratio

  29. Brad Morantz Clustering This can be in N-space or hyperspace Each dimension is a variable Distance can be mahalanobus, Euclidian, or other metric Each Cluster

  30. Brad Morantz Trees Poor No purchase Wealthy Bald Brand A Poor Hair Brand V Over 35 Wealthy Brand G Brand C Long hair Under 35 Brand P Short hair SUV Non SUV Brand L

  31. Brad Morantz Recursive Partitioning Algorithms • Recursively make choice at node purer • Adding another node = another variable • Add until desired accuracy is met • CART • Classification And Regression Trees • Reducing variance at node makes cluster tighter • ID3/C4.5 • Bayes Classification • Reduce entropy/disorganization

  32. Brad Morantz Statistical Processes • Plot it and look at plot • Step wise regression • Significant variables • T value • Eigenvalue • Pearson correlation coefficient • Cluster Analysis

  33. Brad Morantz Bayes Classification • Conditional Probability • P(A|B) = P(A∩B) / P(B) • Independence • If P(A|B) = P(A), then A independent B • If not independent, then B contributes • Compare P(A|B) to P(A|C) to ……. • Can rank variables to see how much knowledge each adds

  34. Brad Morantz Required steps • Build & maintain a database • Data formatting • Data cleansing & anomaly detection • Data visualization • Currently need human expert to evaluate

  35. Brad Morantz Data Preparation • 75% of effort and time • Can mean difference between success and failure • Can greatly affect accuracy

  36. Brad Morantz Data Preparation Example • Mathematical relationships • Weight of animal in shell > animal only • Ratios • More than possible quantity in container • Impossible answers • Out of range answers • Data entry errors • Etc

  37. Brad Morantz Why now? • Rapidly decreasing cost of data storage • Increased ease of collecting data • Networks • Internet • Intranet • Store scanners • Computerized lifestyle • Heavy use of credit/debit cards • Store customer “value” cards • More & cheaper computational power

  38. Brad Morantz Future • Incorporation of background & associated knowledge • More exacting data sources • Hybrid systems • Mixed media use

  39. Brad Morantz References • Leo Breiman • Tom M Mitchell & Bruce Buchanan • Toshinori Munakata • Quinlan • Clark & Niblett

More Related