Fast Algorithms For Mining Association Rules

1. Fast Algorithms For Mining Association Rules By Rakesh Agrawal and R. Srikant Presented By: Chirayu Modi

2. What is Data mining ? • Data mining is a set of techniques used in an automated approach to exhaustively explore and bring to the surface complex relationships in very large datasets.“ • “…is the process of discovering interesting knowledge...from large amounts of data stored in databases, data warehouses, and other information repositories.“ • Some method includes: • Classification and Clustering • Association • Sequencing

3. Association Rules • Association rules are (local) patterns, which model dependencies between attributes. • Finding frequent patterns, associations, correlations, or causal structures among sets of items or objects in transaction databases, relational databases, and other information repositories. • Applications: • Basket data analysis, cross-marketing, catalog design, loss-leader analysis, clustering, classification, etc • Rule form: “Body ® Head [support, confidence]”. • Examples: • buys(x, “beers”) ® buys(x, “chips”) [0.5%, 60%] • major(x, “CS”) and takes(x, “DB”) ® grade(x, “A”) [1%, 75%]

4. Association Rule: Basic Concepts • Given: (1) database of transactions, (2) each transaction is a list of items (purchased by a customer in a visit) • Find: all rules that correlate the presence of one set of items with that of another set of items • E.g., 98% of people who purchase tires and auto accessories also get automotive services done • Applications • *  Maintenance Agreement (What the store should do to boost Maintenance Agreement sales) • Home Electronics  * (What other products should the store stocks up?) • Attached mailing in direct marketing

5. Mining Association Rules—An Example Min. support 50% Min. confidence 50% For rule AC: support = support({AC}) = 50% confidence = support({AC})/support({A}) = 66.6% The Apriori principle: Any subset of a frequent itemset must be frequent

6. Mining Frequent Itemsets: the Key Step • Find the frequent itemsets: the sets of items that have minimum support • A subset of a frequent itemset must also be a frequent itemset • i.e., if {AB} isa frequent itemset, both {A} and {B} should be a frequent itemset • Iteratively find frequent itemsets with cardinality from 1 to k (k-itemset) • Use the frequent itemsets to generate association rules.

7. Mining algorithm • IDEA: It relies on the ”a priori” or downward closure property: if an itemset has minimum support (frequent itemset) then every subset of itemset also has minimum support. • In the first pass the support for each item is counted and the large itemsets are obtained. • In the second pass the large itemsets obtained from the first pass are extended to generate new itemsets called candidate itemsets. • The support of the candidate itemsets is measured and large itemsets are obtained. • This is repeated till no large itemsets can be formed.

8. AIS algorithm • Two concepts are • Extension of an itemset. • Determining what should be in the candidate itemset. • In case of the AIS, candidate sets are generated on the fly. New candidate item sets are generated by extending the large item sets that were generated in the previous pass with other items in the transaction.

9. SETM • The implementation is based on expressing the algorithm in the form of SQL queries. • The 2 steps are: • Generating the candidate itemsets using join operations. • Generating the support counts and determining the large itemsets. • In case of the SETM too, the candidate sets are generated on the fly. New candidate item sets are generated by extending the large item sets that were generated in the previous pass with other items in the transaction.

10. Drawbacks of AIS and SETM • They were very slow. • Generated large number of itemsets with the support/confidence lower than the user specified minimum support/confidence. • Makes a number of passes over the database. • All aspects of data mining cannot be represented using SETM algorithm.

11. Apriori algorithm • Candidate itemsets were generated from large itemsets of previous pass without considering the database. • The large itemsets of the previous pass were extended to get the new candidate itemsets. • Pruning was done using the fact that any subset of a frequent itemset should be frequent. • Step 1 - discover all frequent items that have support above the minimum support required. • Step 2 - Use the set of frequent items to generate the association rules that have high enough confidence

12. Apriori Candidate Generation • Monotonicity Property: All subset of a frequent set are frequent • Given Lk-1, Ck can be generated in two steps: • Join: Join Lk-1 with Lk-1, with the join condition that the first k-1 items should be the same • Prune: delete all candidates whose support is lower than the minimum support specified

13. The Apriori Algorithm • Pseudo-code: Ck: Candidate itemset of size k Lk : frequent itemset of size k L1 = {frequent items}; for(k = 1; Lk !=; k++) do begin Ck+1 = candidates generated from Lk; for each transaction t in database do increment the count of all candidates in Ck+1 that are contained in t Lk+1 = candidates in Ck+1 with min_support end returnkLk;

14. Apriori candidate generation (join step) • The Apriori-generation function takes as argument F(k-1), the set of all frequent (k-1)-item sets. It returns a superset of the set of all frequent k-item sets. The function works as follows: First, in the join step, we join F(k-1) with F(k-1): • insert into C(k) select p.item(1), p.item(2),... p.item(k-1), q.item(k-1) from F(k-1) as p, F(k-1) as qwhere p.item(1) = q.item(1),...,p.item(k-2) = q.item(k-2), p.item(k-1) < q.item(k-1)

15. The prune step • We delete all the item sets c in C(k) such that some (k-1)-subset of c is not in F(k-1): • for each item sets c in C(k) do • for each (k-1)-subsets s of c do • if (s not in F(k-1)) then • delete c from C(k); • Any subset of a frequent item set must be frequent • Lexicographic order of items is assumed!

16. The Apriori Algorithm — Example

17. Buffer Management • In the candidate generation phase of pass k , we need storage for large itemsets Lk-1 and the candidate itemsets Ck. In the counting phase, we need storage for Ck and at least one page to buffer the database transactions. (Ct is a subset of Ck) • Lk fits in memory and Ck does not: generate as many Ck as possible, scan database and count support and write Fk to disk. Delete small itemsets. Repeat until all of Fk is generated for that pass. • Lk-1 does not fit in memory: externally sort Lk-1. Bring into memory Lk-1 items in which the first k-2 items are the same. Generate Candidate itemsets. Scan data and generate Fk. Unfortunately, pruning cannot be done.

18. CORRECTNESS • Ck is a superset of Fk. • Ck is a superset of Fk by the way Ck is generated. • Subset pruning is based on the monotonicity property and every item pruned is guaranteed not be large. • Hence, Ck is always a superset of Fk.

19. AprioriTid Algorithm • It is similar to the Apriori Algorithm and uses Apriori-gen function to determine the candidate sets. • But the basic difference is that for determining the support , the database is not used after the first pass. • Rather a set C’k is used for this purpose • Each member of C’k is of the form <TID, {Xk} > where Xk is Potentially large k itemset present in the transaction with the identifier TID • C’1 corresponds to database D.

20. Algorithm AprioriTID • L1 = ( large 1-itemsets); • C1’= database D; • for (k=2; Lk-1 ; k++) do begin • Ck = apriori-gen(Lk-1); //New candidates • Ck’= ; • forall entries t  Ck-1’ do begin • //determine candidate itemsets in Ck contained in the transaction with identifier t.TID Ct ={c  Ck |(c-c[k]) t.set-of-itemsets  (c-c[k-1]) t.set-of-itemsets }; • forall candidates c  Ct do • c.count++; • if (Ct   ) then Ck’+= < t.TID, Ct>; • end; • Lk = {c  Ck | c.count  minsup} • End • Answer = Uk Lk;

21. Buffer management • In the kth pass, AprioriTid needs memory for Lk-1 and Ck-1 during candidate generation. • During the counting phase, it needs memory for Ck-1 , Ck , and a page for Ck-1 ‘ and Ck ‘. entries in C’k-1 are needed sequentially, but C’k can be written out as generated.

22. Drawbacks of Apriori and AprioriTid • Apriori • Takes longer time for calculating support of candidate itemsets. • For determining the support of the candidate sets the algorithm always looks into every transaction in the database. Hence it takes a longer time (more passes on data) • AprioriTid • During the initial passes the candidate itemsets generated are very large equivalent to the size of the database. Hence the time taken will be equal to that of Apriori. And also it might incur an additional cost if it cannot completely fit into the memory.

23. Experimental results • As the minimum support decreases, the execution times of all the algorithms increase because of increases in the total number of candidate and large itemsets. • Apriori beats SETM by more than an order of magnitude for large datasets. • Apriori beats AIS for all problem sizes, by factors ranging from 2 for high minimum support to more than an order of magnitude for low levels of support. • For small problems, AprioriTid did about as well as Apriori, but performance degraded to about twice as slow for large problems.

24. Apriori Hybrid • Initial pass: Apriori performs better • Later pass: AprioriTid performs better Apriori Hybrid: • Uses Apriori in the initial passes and later shifts to AprioriTid. • Drawback: • An extra cost is incurred when shifting from Apriori to AprioriTid. • Suppose at the end of K th pass we decide to switch from Apriori to AprioriTid. Then in the (k+1) pass, after having generated the candidate sets we also have to add the Tids to C’k+1

25. Is Apriori Fast Enough? — Performance Bottlenecks • The core of the Apriori algorithm: • Use frequent (k – 1)-itemsets to generate candidate frequent k-itemsets • Use database scan and pattern matching to collect counts for the candidate itemsets • The bottleneck of Apriori: candidate generation • Huge candidate sets: • 104 frequent 1-itemset will generate 107 candidate 2-itemsets • To discover a frequent pattern of size 100, e.g., {a1, a2, …, a100}, one needs to generate 2100  1030 candidates. • Multiple scans of database: • Needs (n +1 ) scans, n is the length of the longest pattern

26. Conclusions • Experimental results were shown to prove that the proposed algorithms outperform AIS and SETM. The performance gap increased with the problem size, and ranged from a factor of three for small problems to more than an order of magnitude for large problems. • Best features of the two proposed algorithms can be combined into a hybrid algorithm which then becomes an algorithm of choice. Experiments demonstrate the feasibility of using AprioriHybrid in real applications involving very large databases.

27. Future Scope In future authors plan to extend this work along the following dimensions: Multiple taxonomies (is-a hierarchies) over items are often available. An example of such a hierarchy is that a dish washer is a kitchen appliance is a heavy electric appliance, etc. Authors are interested in discovering the association rules that use such hierarchies. Authors did not consider the quantities of the items bought in a transaction, which are useful for some applications. Finding such rules needs further work.

28. Questions & Answers