Data Mining Project

1. Data Mining Project Team 4 Christine Dorothy - HT062300W DjokoJoewono - HT062137Y Kang TiongMeng - HT072164H Ng Hock Leong Tommy - HT072172H Wang Long - HT072109B HooSoo Yean

2. Project Overview • Mail Order Company in USA • Would like to find out if there is a way • To reduce mailing cost • By analyzing the past data

3. CRISP Process Overview

4. Business Understanding • Business Objectives: • To find out which customers that are good candidate to purchase products • To explore the data to determine company’s valuable customers

5. Business Understanding • Assess the Situation: • One CSV file from 3 data sources • Census Group A • Census Group B • Tax Filers • Personnel • Six MTech Students • Minimum experience in data mining • Software • MS Excel • Clementine, Data Scope

6. Business Understanding • Data Mining Goals • Predict which variables affects customer buying decision • Build models and compare the cost against randomly-chosen customers • Suggest a model to achieve >1% mailing response

7. Project Plan

8. Data Understanding • First Insights Discovery • Total record is 2158 • Distribution by Objective • Distribution by Gender

9. Data Understanding • Data Quality Problems • Some columns are normalized others not • All values are number, harder to visualize • Many data is incomplete • Missing recency, no of transactions and dollars of spending data for individual products

10. Data Understanding • Describe Data • Gross properties of data • The data is extracted from a larger set with respond rate of ~1%.All 1079 responders and 1079 randomly chosen non-responders • Relationship between attributes • firstmonth and tenure have a linear relationship, Thus tenure can be omited.

11. firstmonth Vs tenure

15. Histogram of product6

16. Data Preparation • Select Data • Variables chosen • Clean Data • Some normalizations • Construct Data • Chose the variables as input • Data Transformation • Rescaling • Derive new variables

17. Data Cleaning Reduce redundancy caused by data integration Replace lowincome and highincome with IncomeGroup. Replace gender1,gender2 and gender3 with Gender. Discard V171 Total taxfilers with unemployment benefits Discard V175, V181,V184, V190,V193,V196. they equal to male data plus female data

18. Data Transformation Rescaling Log() of totalspend and totaltrans to reduce effect of large variables Derive Data Derive ActAccInMostRecMon from product recency data(no of active accounts in most recent month) Derive the ratio of low taxfiler income from V156-V163 Value=V156/sum(V156:V163) Convert value to 5 categories.

19. Data Transformation Histogram of new variable with Objective overlaid

20. Gender Vs IncomeGroup

21. Multiplot for p16

22. totaltranslog vs totalspendlog

23. totalspendlog over totaltranslog(Sum)

24. Histogram of ActAccInMostRecMon

25. Census A Languages

26. Census A Languages (cont’d) Inverse correlation between English and French speaking regions No region with significant Tagalog, Spanish or other language-speaking populations Can probably discard amtspanish, amttagalog, amtsingres, amtengnon, amtmultilin Cluster/segment English/French areas

27. Census A vs B Languages

28. Census A vs B Languages

29. Census A vs B Languages Linear relationship for English and French across Census A & B Can merge amtenglish and bhlenglish Can merge amtfrench and bhlfrench

30. Acflonepar vs bfslonepar

31. Acflonepar vs bfslonepar Linear relationship Merge acflonepar & bfslonepar Filter out noisy data

32. Anfamrel – living with relatives

33. Anfamrel – living with relatives Most data below 0.1 Objective remains constant throughout Not important to business objective – discard

34. afem40to44

35. afem40to44 Lack of data from other age groups Very specific targeted marketing to 40-44 females group Normalize values from 0 to 0.1 if necessary Objective improves as proportion increases

36. afp1child

37. afp1child Objective clearly improves when afp1child is on lower end of normal curve

38. acfwchcom 7 regions with acfwchcom = 0.19 and objective = 1

39. Acftotmar vs acfhuswife

40. Acftotmar vs acfhuswife Most regions have above 60% married couples, assuming normalized data Acftotmar and acfhuswife mirror one another Can discard either field Filter noisy data Categorical : lone-parent and husband-wife

41. Census B Data Understanding As the other cencus and taxfiler data, these data represents the distribution of the region.

42. Period of Construction • There is a similar trend, the number of construction between the two period is more or less the same number. • The sample population only represents a small number of people of construction in the region.

43. Maintenance & Repair • Those who does regular maintenance does not have major nor minor repair

44. Maintenance & Repair • Those who has major repair, tend to have less minor repair.

45. Ethnic Origin • These sample population represents majority of the English or British ethnic origin in the region. • Those who has British ethnic origin also has English ethnic origin. • Those who has English ethnic origin is less than British ethnic origin.

46. Ethnic Origin • This data only represents a very low number of people who is French ethnic origin.

47. Household vs Family Income • Both have the same trend, some who doesn’t answer for family income, answered for household income

48. Labour Force Variable • Both of them has the same description. Need to check which one is which.

49. Birth City • The population sample is mostly locals