1 / 15

CSI5388 Practical Recommendations

CSI5388 Practical Recommendations. Context for our Recommendations I. This discussion will take place in the context of the following three questions: I have created a new classifier for a specific problem. How does it compare to other existing classifiers on this particular problem?

eduardoa
Download Presentation

CSI5388 Practical Recommendations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CSI5388Practical Recommendations

  2. Context for our Recommendations I This discussion will take place in the context of the following three questions: • I have created a new classifier for a specific problem. How does it compare to other existing classifiers on this particular problem? • I have designed a new classifier, how does it compare to existing classifiers on benchmark data? • How do various classifiers fare on benchmark data or on a single new problem?

  3. Context for our Recommendations II These three questions can be translated into the four different situations: • Situation 1: Comparison of a new classifier to generic ones for a specific problem • Situation 2: Comparison of a new classifier to generic ones on generic problems • Situation 3: Comparison of generic classifiers on generic domains • Situation 4: Comparison of generic classifiers on a specific problem

  4. Selecting learning algorithms I • The general strategy is to try to select classifiers that are more likely to succeed on the task at hand. • Situation 1: Select generic classifiers with a good chance of success at the particular task. • E.g., For high dimensionality problem: use SVM as a generic classifier • E.g., For class imbalanced problem: use SMOTE as a generic classifier, etc. • Situation2: Different from Situation 1 in that not specific problem is targeted. So, choose generic classifiers that are generally accurate and stable across domains • E.g., Random Forests, SVMs, Bagging

  5. Selecting learning algorithms II • Situation 3: Different from Situations 1 and 2. This time, we are interested in finding the strengths and weaknesses of various algorithms on different problems. So, select various well-known and well-used algorithms. Not necessarily the best algorithms overall. • E.g., Decision Trees, Neural Networks, Naïve Bayes, Nearest Neighbours, SVMs, etc. • Situation 4: reduces to Case 1 where what matters is the search for an optimal classifier or to Case 3, where the purpose is of a more general and scientific nature.

  6. Selecting Data Sets I • The selection of data sets is different in the cases of Situations 1 and 4 and Situations 2 and 3. • Situations 1 and 4: We distinguish between two cases: • Case 1: There is just one data set of interest – Just use this data set. • Case 2: We are considering a class of data sets (e.g., data sets for text categorization). In this case, we should look at Situations 2 and 3, since data sets in the same class can have different characteristics (e.g., noise, class imbalances, etc). The only difference is that the domains in this class will be more closely related than those in a wider study of the kind considered in Situations 2 and 3.

  7. Selecting Data Sets II • Situations 2 and 3: The first thing that we need to do is determine what the exact purpose of the study is. • Case 1: To test a specific characteristic of a new algorithm or of various algorithms (e.g., their resilience to noise) – Select domains presenting the same characteristics • Case 2: To test the general performance of a new algorithm or of various algorithms on a variety of domains with different characteristics — Select varied domains, but watch the way in which you report the results. There may be a lot of variance, from classifier to classifier and type of domain to type of domain. It will be best to cluster the kinds of domains on which classifiers excel or do poorly and report the results on a cluster by cluster basis.

  8. Selecting Data Sets III • Situations 2 and 3 (Cont’d): Three questions remain: • Question 1: How many data sets are necessary / desirable? • Question 2: Where can we get these data sets? • Question 3: How do we select data sets from those available?

  9. Selecting Data Sets IV Situations 2 & 3: How many data sets? The number of domains necessary depends on the variance in the performance of the classifiers. As a rule of thumb, 3 to 5 domains within the same category of domains are desirable to begin with. Note: As domains get added, the question raised by [Salzberg, 1997] and [Jensen, 2001] regarding the multiplicity effect should be considered. Situations 2 & 3: Where can we get these data sets? • UCI Repository for machine learning or other repositories (but the collections may not be representative of reality). • Directly from the Web (but gathering a cleaning a data collection is extremely time consuming) • Artificial data sets (easy to build, unlimited in size, but too far removed from reality) • Real-world inspired artificial data (real-world data sets artificially augmented. Easy to build, closer to reality)

  10. Selecting Data Sets V Situations 2 & 3: How do we select data sets from those available? • Select all those that are available and meet the constraints of the algorithms that are under study. For example, the UCI repository contain many data sets, but only a subset of these are multi-class, only a subset has nominal attributes only, only a subset has no missing attributes, and so on. • In order to increase the number of domains available for use by researchers or practitioners of Data Mining, some amendments to the data sets can be made to make as many data sets as possible conform to the requirements of the classifiers.

  11. Selecting performance measures • Cases 2 and 3: Caruana and Niculescu-Mizil, 2004 suggest that the Root mean Squared error is the best general-purpose method since it is the one that is best correlated with the other eight measures that they use. Researchers are, however, encouraged to use a variety of different metrics in order to discover the various strengths and shortcomings of each classifier and each domain more specifically. • Cases 1 and 4: We distinguish between the following cases: • Balanced versus imbalanced domains: ROC • Certainty of the decision matters: B & K • All the classes matter: RMSE • The problem is binary but one class matters more than the other: Precision, Recall, F-measure, Sensitivity, Specificity, Likelihood Ratios.

  12. Selecting an error estimation method and statistical test I • If the size of the data set is large enough (the size of all testing sets is, at least, 30) and if the statistics of interest to the user is parameterizable: cross-validation can be tried (but see the next slide). • If the data set is particularly small, i.e., if some of the testing sets contain fewer than 30 examples: say, if it contains fewer than 30, or so samples: Bootstrapping or Randomization. • If the statistics of interest does not have statistical tests associated with it: Bootstrapping or Randomization.

  13. Selecting an error estimation method and statistical test II • Question: How can one see whether cross-validation is appropriate for his/her purposes? • 2 ways: • Visual: plot the distribution and check its shape visually • Apply a Hypothesis Test designed to see if the distribution is normal or not. (e.g. Chi squared goodness of fit, Kolmogorov-Smirnov goodness of fit, etc.) • Since no practical distribution will be exactly Normal, we must also look into the robustness of the various statistical method considered. The t-test is quite robust against the normality assumption. • If the distribution is far from normal non-parametric tests must be used.

  14. Selecting an error estimation method and statistical test III • The robustness of a procedure is important since that will ensure that the reported significance level is close to the true one. • However, Robustness does not answer the question of whether efficient use is made of the data so that a false null hypothesis can be rejected. • Power should be considered • The power of a test depends on some intrinsic nature of that test, but also on the shape and size of the population to which it is applied. • Example: Parametric tests based on the normality assumption are generally as powerful or more powerful than non-parametric tests based on ranks in the case of distribution functions with lighter tails than the normal distribution.

  15. Selecting an error estimation method and statistical test IV • But: Parametric tests based on the normality assumption are less powerful than non-parametric ones in the case where the tails of the distribution are heavier than those of the normal distribution (An important kind of data presenting such distributions are data containing outliers). • Note that the relative power of parametric and non-parametric tests does not change as a function of sample size, even if a test is asymptotically distribution free (i.e., if it becomes more and more robust as the sample size increases).

More Related