Quiz 3: Mean: 9.2 Median: 9.75 Go over problem 1

1. Quiz 3:Mean: 9.2Median: 9.75Go over problem 1

2. Go over Adaboost examples

3. Fix to C4.5 data formatting problem?

4. Quiz 4

5. Alternative simple (but effective) discretization method(Yang & Webb, 2001) Let n = number of training examples. For each attribute Ai , create bins. Sort values of Ai in ascending order, and put of them in each bin. Don’t need add-one smoothing of probabilities This gives good balance between discretization bias and variance.

6. Alternative simple (but effective) discretization method(Yang & Webb, 2001) Let n = number of training examples. For each attribute Ai , create bins. Sort values of Ai in ascending order, and put of them in each bin. Don’t need add-one smoothing of probabilities This gives good balance between discretization bias and variance. Humidity: 25, 38, 50, 80, 93, 98, 98,, 99

7. Alternative simple (but effective) discretization method(Yang & Webb, 2001) Let n = number of training examples. For each attribute Ai , create bins. Sort values of Ai in ascending order, and put of them in each bin. Don’t need add-one smoothing of probabilities This gives good balance between discretization bias and variance. Humidity: 25, 38, 50, 80, 93, 98, 98,, 99

8. Alternative simple (but effective) discretization method(Yang & Webb, 2001) Let n = number of training examples. For each attribute Ai , create bins. Sort values of Ai in ascending order, and put of them in each bin. Don’t need add-one smoothing of probabilities This gives good balance between discretization bias and variance. Humidity: 25, 38, 50, 80, 93, 98, 98,, 99

9. Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifer(P. Domingos and M. Pazzani) Naive Bayes classifier is called “naive” because it assumes attributes are independent of one another.

10. This paper asks: why does the naive (“simple”) Bayes classifier, SBC, do so well in domains with clearly dependent attributes?

11. Experiments • Compare five classification methods on 30 data sets from the UCI ML database. SBC = Simple Bayesian Classifier Default = “Choose class with most representatives in data” C4.5 = Quinlan’s decision tree induction system PEBLS = An instance-based learning system CN2 = A rule-induction system

12. For SBC, numeric values were discretized into ten equal-length intervals.

14. Number of domains in which SBC was more accurate versus less accurate than corresponding classifier Same as line 1, but significant at 95% confidence Average rank over all domains (1 is best in each domain)

15. Measuring Attribute Dependence They used a simple, pairwise mutual information measure: For attributes Am and An,dependence is defined as where AmAnis a “derived attribute”, whose values consist of the possible combinations of values of Am and An Note: If Am and An are independent, then D(Am, An | C) = 0.

16. Results: (1) SBC is more successful than more complex methods, even when there is substantial dependence among attributes. (2) No correlation between degree of attribute dependence and SBC’s rank. But why????

17. An Example • Let C = {+, −}, and attributes = {A, B, C}. • Let P(+) = P(−) = 1/2. • Suppose A and C are completely independent, and A and B are completely dependent (e.g., A = B). • Optimal classification procedure:

18. This leads to the followingOptimal Classifier conditions: If P(A|+) P(C|+) > P(A | −) P(C| −) then class = + = else class = − • SBC conditions If P(A|+)2 P(C|+) > P(A | −)2 P(C| −) then class = + else class = −

19. In the paper, the authors use Bayes Theorem to rewrite these conditions, and plot the “decision boundaries” for the optimal classifier and for the SBC. + p = P(+ |A) q= P(+ | C) Optimal SBC −

20. Even though A and B are completely dependent, and the SBC assumes they are completely independent, the SBC gives the optimal classification in a very large part of the problem space! But why?

21. Explanation: SupposeC= {+,−} are the possible classes. Letxbe a new example with attributes <a1, a2, ..., an>. What the naive Bayes classifier does is calculates two probabilities, and returns the class that has the maximum probability givenx.

22. The probability calculations are correct only if the independence assumption is correct. • However, the classification is correct in all cases in which the relative ranking of the two probabilities, as calculated by the SBC, is correct! • The latter covers a lot more cases than the former. • Thus, the SBC is effective in many cases in which the independence assumption does not hold.

23. More on Bias and Variance

24. From http:// eecs.oregonstate.edu/~tgd/talks/BV.ppt Bias

25. From http:// eecs.oregonstate.edu/~tgd/talks/BV.ppt Variance

26. From http:// eecs.oregonstate.edu/~tgd/talks/BV.ppt Noise

27. From http:// eecs.oregonstate.edu/~tgd/talks/BV.ppt Sources of Bias and Variance • Bias arises when the classifier cannot represent the true function – that is, the classifier underfits the data • Variance arises when the classifier overfits the data • There is often a tradeoff between bias and variance

28. From knight.cis.temple.edu/~yates/cis8538/.../intro-text-classification.ppt Bias-Variance Tradeoff As a general rule, the more biased a learning machine, the less variance it has, and the more variance it has, the less biased it is.

29. From: http://www.ire.pw.edu.pl/~rsulej/NetMaker/index.php?pg=e06

30. From knight.cis.temple.edu/~yates/cis8538/.../intro-text-classification.ppt Bias-Variance Tradeoff As a general rule, the more biased a learning machine, the less variance it has, and the more variance it has, the less biased it is. Why?

31. From http:// eecs.oregonstate.edu/~tgd/talks/BV.ppt SVM Bias and Variance • Bias-Variance tradeoff controlled by s • Biased classifier (linear SVM) gives better results than a classifier that can represent the true decision boundary!

32. From http:// eecs.oregonstate.edu/~tgd/talks/BV.ppt Effect of Boosting • In the early iterations, boosting is primary a bias-reducing method • In later iterations, it appears to be primarily a variance-reducing method

33. Bayesian NetworksReading: S. Wooldridge, Bayesian belief networks(linked from class website)

34. A patient comes into a doctor’s office with a fever and a bad cough. Hypothesis space H: h1: patient has flu h2: patient does not have flu Data D: coughing= true, fever = true,, smokes = true

35. Naive Bayes Cause smokes cough fever flu Effects

36. Full joint probability distribution smokes Sum of all boxes is 1. In principle, the full joint distribution can be used to answer any question about probabilities of these combined parameters. However, size of full joint distribution scales exponentially with number of parameters so is expensive to store and to compute with.  smokes

37. Bayesian networks • Idea is to represent dependencies (or causal relations) for all the variables so that space and computation-time requirements are minimized. smokes cough fever flu “GraphicalModels”

38. cough Conditional probability tables for each node flu smoke smoke flu flu smoke cough fever fever flu

39. Semantics of Bayesian networks • If network is correct, can calculate full joint probability distribution from network. where parents(Xi) denotes specific values of parents of Xi.

40. Example • Calculate

41. Another (famous, though weird) Example Rain Wet grass Question: If you observe that the grass is wet, what is the probability it rained?

43. Sprinkler Rain Wet grass Question: If you observe that the sprinkler is on, what is the probability that the grass is wet? (Predictive inference.)

45. Question: If you observe that the grass is wet, what is the probability that the sprinkler is on? (Diagnostic inference.) Note that P(S) = 0.2. So, knowing that grass is wet increased probability that sprinkler is on.

46. Now assume the grass is wet and it rained. What is the probability that the sprinkler was on? Knowing that it rained decreases the probability that the sprinkler was on, given that the grass is wet.

47. Cloudy Sprinkler Rain Wet grass Question: Given that it is cloudy, what is the probability that the grass is wet?

49. In general... • If network is correct, can calculate full joint probability distribution from network. where parents(Xi) denotes specific values of parents of Xi. But need efficient algorithms to do this (e.g., “belief propagation”, “Markov Chain Monte Carlo”).

50. Complexity of Bayesian Networks For n random Boolean variables: • Full joint probability distribution: 2n entries • Bayesian network with at most k parents per node: • Each conditional probability table: at most 2kentries • Entire network: n 2k entries