Adaptive Fraud Detection

1. Adaptive Fraud Detection by Tom Fawcett and Foster Provost Presented by: Lara Nargozian updated from last 3 year’s presentation by Adam Boyer, Yunfei Zhao and Ahmen Abdeen Hamed

2. Why? • Solving real-world problems that are very important to each and everyone of us • Provide a framework that can be adapted to solve similar problems • Use Data Mining algorithms and techniques learned this semester • Rule Learning • Classification • Fun to learn about

3. Outline • Problem Description • Cellular cloning fraud problem • Why it is important • Current strategies • Construction of Fraud Detector • Framework • Rule learning, Monitor construction, Evidence combination • Experiments and Evaluation • Data used in this study • Data preprocessing • Comparative results • Conclusion • Exam Questions

4. The Problem • How to detect suspicious changes in user behavior to identify and prevent cellular fraud • Non-legitimate users, aka bandits, gain illicit access to a legitimate user’s, or victim’s, account • Solution useful in other contexts • Identifying and preventing credit card fraud, toll fraud, and computer intrusion

5. Cellular Fraud - Cloning • Cloning Fraud • A kind of Superimposition fraud.(parasite) • Fraudulent usage is superimposed upon ( added to ) the legitimate usage of an account. • Causes inconvenience to customers and great expense to cellular service providers.

6. Cellular communications andCloning Fraud • Mobile Identification Number (MIN) and Electronic Serial Number (ESN) • Identify a specific account • Periodically transmitted unencrypted whenever phone is on • Cloning occurs when a customer’s MIN and ESN are programmed into a cellular phone not belonging to the customer • Bandit can make virtually unlimited, untraceable calls at someone else’s expense

7. Interest in reducing Cloning Fraud • Fraud is detrimental in several ways: • Fraudulent usage congests cell sites • Fraud incurs land-line usage charges • Cellular carriers must pay costs to other carriers for usage outside the home territory • Crediting process is costly to carrier and inconvenient to the customer

8. Strategies for dealing with cloning fraud • Pre-call Methods • Identify and block fraudulent calls as they are made • Validate the phone or its user when a call is placed • Post-call Methods • Identify fraud that has already occurred on an account so that further fraudulent usage can be blocked • Periodically analyze call data on each account to determine whether fraud has occurred.

9. Pre-call Methods • Personal Identification Number (PIN) • PIN cracking is possible with more sophisticated equipment. • RF Fingerprinting • Method of identifying phones by their unique transmission characteristics • Authentication • Reliable and secure private key encryption method. • Requires special hardware capability • An estimated 30 million non-authenticatable phones are in use in the US alone (in 1997)

10. Post-call Methods • Collision Detection • Analyze call data for temporally overlapping calls • Velocity Checking • Analyze the locations and times of consecutive calls • Disadvantage of the above methods • Usefulness depends upon a moderate level of legitimate activity

11. Another Post-call Method( Main focus of this paper ) • User Profiling • Analyze calling behavior to detect usage anomalies suggestive of fraud • Works well with low-usage customers • Good complement to collision and velocity checking because it covers cases the others might miss

12. Sample Frauded Account

13. The Need to be Adaptive • Patterns of fraud are dynamic – bandits constantly change their strategies in response to new detection techniques • Levels of fraud can change dramatically from month-to-month • Cost of missing fraud or dealing with false alarms change with inter-carrier contracts

14. Automatic Construction of Profiling Fraud Detectors

15. One Approach • Build a fraud detection system by classifying calls as being fraudulent or legitimate • However there are two problems that make simple classification techniques infeasible.

16. Problems with simple classification • Context • A call that would be unusual for one customer may be typical for another customer (For example, a call placed from Brooklyn is not unusual for a subscriber who lives there, but might be very strange for a Boston subscriber. ) • Granularity (over fitting?) • At the level of the individual call, the variation in calling behavior is large, even for a particular user.

17. In Summary: Learning The Problem • Which phone call features are important? • How should profiles be created? • When should alarms be raised?

18. Proposed Detector Constructor Framework (DC-1)

19. DC-1 Processing Account-Day Example

20. DC-1 Fraud Detection Stages • Stage 1: Rule Learning • Stage 2: Profile Monitoring • Stage 3: Combining Evidence

21. Rule Learning – the 1st stage • Rule Generation • Rules are generated locally based on differences between fraudulent and normal behavior for each account • Rule Selection • Then they are combined in a rule selection step

22. Rule Generation • DC-1 uses the RL program to generate rules with certainty factors above user-defined threshold • For each Account, RL generates a “local” set of rules describing the fraud on that account. • Example: (Time-of-Day = Night) AND (Location = Bronx)  FRAUD Certainty Factor = 0.89

23. Rule Selection • Rule generation step typically yields tens of thousands of rules • If a rule is found in ( or covers ) many accounts then it is probably worth using • Selection algorithm identifies a small set of general rules that cover the accounts • Resulting set of rules is used to construct specific monitors

24. Rule Selection and Covering Algorithm

25. Profiling Monitors – the 2nd stage Monitors have 2 distinct steps - • Profiling step: • Monitor is applied to an account’s normal usage to measure the account‘s normal activity. • Statistics are saved with the account. • Use step: • A monitor processes a single account-day, • References the normalcy measure from profiling • Generates a numeric value describing how abnormal the current account-day is.

26. Most Common Monitor Templates • Threshold • Standard Deviation

27. Threshold Monitors

28. Standard Deviation Monitors

29. Comparing the same standard deviation monitor on two accounts

30. Example for Standard Deviation • Rule • (TIME­OF­DAY = NIGHT) AND (LOCATION = BRONX) FRAUD • Profiling Step • the subscriber called from the Bronx an average of 5 minutes per night with a standard deviation of 2 minutes. At the end of the Profiling step, the monitor would store the values (5,2) with that account. • Use step • if the monitor processed a day containing 3 minutes of airtime from the Bronx at night, the monitor would emit a zero; if the monitor saw 15 minutes, it would emit (15 - 5)/2 = 5. This value denotes that the account is five standard deviations above its average (profiled) usage level.

31. Combining Evidence from the Monitors – the 3rd stage • Weights the monitor outputs and learns a threshold on the sum to produce high confidence alarms • DC-1 uses Linear Threshold Unit (LTU) • Simple and fast • Enables good first-order judgment • A Feature selection process is used to • Choose a small set of useful monitors in the final detector • Some rules don’t perform well when used in monitors, some overlap • Forward selection process chooses set of useful monitors

32. Final Output of DC-1 • Detector that profiles each user’s behavior based on several indicators • An alarm when sufficient evidence of fraudulent activity

33. Data used in the study

34. Data Information • Four months of phone call records from the New York City area. • Each call is described by 31 original attributes • Some derived attributes are added • Time-Of-Day (MORNING, AFTERNOON, TWILIGHT, EVENING, NIGHT) • To-Payphone • Each call is given a class label of fraudulent or legitimate.

35. Data Cleaning • Eliminated credited calls made to destinations/numbers that are not in the created block • The destination number must be only called by the legitimate user. • Days with 1-4 minutes of fraudulent usage were discarded. • May have credited for other reasons, such as wrong number • Call times were normalized to Greenwich Mean Time for chronological sorting

36. Data Description • Once the monitors are created and accounts profiled, the system transforms raw call data into a series of account-days using the monitor outputs as features • Selected for Profiling, training and testing: • 3600 accounts that have at least 30 fraud-free days of usage before any fraudulent usage. • Initial 30 days of each account were used for profiling. • Remaining days were used to generate 96,000 account-days. • Distinct training and testing accounts:10,000 account-days for training; 5000 for testing • 20% fraud days and 80% non-fraud days

37. Experiments and Evaluation

38. Output of DC-1 components • Rule learning: 3630 rules • Each covering at least two accounts • Rule selection: 99 rules • 2 monitor templates yielding 198 monitors • Final feature selection: 11 monitors

39. The Importance Of Error Cost • Classification accuracy is not sufficient to evaluate performance • Should take misclassification costs into account • Estimated Error Costs: • False positive(false alarm): $5 • False negative (letting a fraudulent account-day go undetected): $0.40 per minute of fraudulent air-time • Factoring in error costs requires second training pass by LTU

40. Alternative Detection Methods • Collisions + Velocities • Errors almost entirely due to false negatives • High Usage – detect sudden large jump in account usage • Best Individual DC-1 Monitor • (Time-of-day = Evening) ==> Fraud • SOTA - State Of The Art • Incorporates 13 hand-crafted profiling methods • Best detectors identified in a previous study

41. DC-1 Vs. Alternatives

42. Shifting Fraud Distributions • Fraud detection system should adapt to shifting fraud distributions To illustrate the above point - • One non-adaptive DC-1 detector trained on a fixed distribution ( 80% non-fraud ) and tested against range of 75-99% non-fraud • Another DC-1 was allowed to adapt (re-train its LTU threshold) for each fraud distribution • Second detector was more cost effective than the first

44. Conclusion • DC-1 uses a rule­learning program to uncover indicators of fraudulent behavior from a large database of customer transactions. • Then the indicators are used to create a set of monitors, which profile legitimate customer behavior and indicate anomalies. • Finally, the outputs of the monitors are used as features in a system that learns to combine evidence to generate high­confidence alarms.

45. Conclusion • Adaptability to dynamic patterns of fraud can be achieved by generating fraud detection systems automatically from data, using data mining techniques • DC-1 can adapt to the changing conditions typical of fraud detection environments • Experiments indicate that DC-1 performs better than other methods for detecting fraud

46. Exam Questions

47. Question 1 • What are the two major fraud detection categories, differentiate them, and where does DC-1 fall under? • Pre Call Methods • Involves validating the phone or its user when a call is placed. • Post Call Methods – DC1 falls here • Analyzes call data on each account to determine whether cloning fraud has occurred.

48. Question 2 • Why do fraud detection methods need to be adaptive? • Bandits change their behavior- patterns of fraud dynamic • Levels of fraud varies month-to-month • Cost of missing fraud or handling false alarms changes between inter-carrier contracts

49. Question 3 • What are the two steps of profiling monitors and and what are the two main monitor templates? • Profiling Step: measure an accounts normal activity and save statistics • Use Step: process usage for an account-day to produce a numerical output describing how abnormal activity was on that account-day • Threshold and Standard Deviation monitors.

50. The End. Questions?