1 / 14

Ch19. Evaluation Criteria for BCI Research

Ch19. Evaluation Criteria for BCI Research. Contents. Introduction Criteria for Evaluating Trial-based BCI Data Criteria for Evaluating Self-Paced BCI Data Other criteria. Introduction. The factors affect BCI performance Trial-based (system-paced) <-> Asynchronous mode (self-paced)

ellis
Download Presentation

Ch19. Evaluation Criteria for BCI Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Ch19.Evaluation Criteria for BCI Research

  2. Contents • Introduction • Criteria for Evaluating Trial-based BCI Data • Criteria for Evaluating Self-Paced BCI Data • Other criteria

  3. Introduction • The factors affect BCI performance • Trial-based (system-paced) <-> Asynchronous mode (self-paced) • Type and number of EEG feature • Spectral parameter • Slow cortical potentials • Spatiotemporal parameters • Nonlinear features • Type of classifier • Linear and quadratic discriminate analysis • Support vector machines • Neural networks • Simple threshold detection • Target application

  4. Introduction • The necessity of consistent evaluation criteria • For compare different BCI systems and approaches • Consideration for evaluation criteria • What is being evaluated • Feedback loop • The most frequently used evaluation criterion • Error rate or accuracy • Response speed of BCIs • Evaluation of asynchronous BCI data

  5. Criteria for Evaluating Trial-based BCI Data • The Confusion Matrix • Classification Accuracy and Error Rate • Cohen’s Kappa Coefficient • Mutual Information of a Discrete Output

  6. The Confusion Matrix Correctly classified Incorrectly classified TP: True PositiveTN: True Negative FP: False PositiveFN: False Negative FA: False Activation CR: Correct Rejection

  7. Classification Accuracy and Error Rate • The most widely used evaluation criteria in BCI research • Denoted as ACC for classification ACCuracy, ERR(=1-ACC) for error rate • Can be very easily calculated and interpreted • The accuracy is 100%/M (M: number of class) • The maximum accuracy can never exceed 100% • Some limitation • The off-diagonal values of the confusion matrix are not considered • Classification accuracy of less frequent classes have small weight Correctly classified sample Total number of sample

  8. Cohen’s Kappa Coefficient • Addresses several of the critiques on the accuracy measure • Use the overall agreementpo, and the chance agreement pe Sum of i-th column Sum of i-th row Predicted classes show no correlation with actual classes Perfect classification Different assignment between output and the true classes Standard error of kappa coefficient

  9. Cohen’s Kappa Coefficient • Address several of the criticisms of the accuracy measure • It considers the distribution of the wrong classifications=> i.e., off-diagonal elements of the confusion matrix) • Frequency of occurrence is normalized for each class => class with less samples get the same weight as class with many samples • The standard error of the kappa coefficient easily can be used for comparing whether the results of distinct classification systems have statistically significant differences

  10. Mutual Information of a Discrete Output • Assume following things • BCI system can be modeled as communication channel • Communication theory of Shannon and Weaver (1949) can be applied directly to quantify the information transfer • Farwell and Donchin (1988) • Information transfer for M classes can be calculated as • Problms • The information rate assume an error-free system • This suggestion is not useful for comparing different BCI systems

  11. Mutual Information of a Discrete Output • Based on Pierce (1980), Wolpaw et al. (2000a) • Information transfer rate for M classes an ACC = pois • The formula has following limitations • M selections (classes) are possible • Each class has the same probability • The specific accuracy is same for each class • Each undesired selection must have the same probability of selection • Often these assumption are not fulfilled It does not satisfy 3rd and 4th condition

  12. Mutual Information of a Discrete Output • Random variable X models the user intension • Random variable Y models the classifier output • The entropy H(X) of a discrete random variable is defined as • Nykopp(2001) derived the information transfer for a general confusion matrix Probability to classify xi as yj a priori probability for class xi Mutual information

  13. Criteria for Evaluating Self-paced BCI Data Asynchronous mode BCI • The BCI system is specially designed to produce outputs in response to intentional control • HF-Difference • Hit False difference • H : hit rate, F: false detection rate

  14. Other Criteria • Receiver-Operator Characteristics (ROC) • Correlation Coefficient • Evaluation of Continuous-Input and Continuous-Output Systems • Response time • High accuracy is important • But the response time is also important • Maximum Steepness of the Mutual Information is used in BCI Competition III t0 : time for the cue onset I(t) : continuous mutual information

More Related