Active learning challenge
This presentation is the property of its rightful owner.
Sponsored Links
1 / 27

Active Learning Challenge PowerPoint PPT Presentation


  • 125 Views
  • Uploaded on
  • Presentation posted in: General

Active Learning Challenge . Isabelle Guyon (Clopinet, California)

Download Presentation

Active Learning Challenge

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Active learning challenge

Active Learning Challenge

Isabelle Guyon (Clopinet, California)

Gavin Cawley (University of East Anglia, UK) Olivier Chapelle (Yahhoo!, California) Gideon Dror (Academic College of Tel-Aviv-Yaffo, Israel) Vincent Lemaire (Orange, France) Amir Reza Saffari Azar (Graz University of Technology) Alexander Statnikov (New York University, USA)

http://clopinet.com/al


What is the problem

What is the problem?

http://clopinet.com/al


Labeling data is expensive

Labeling data is expensive

$$$$$

$$

Unlabeled

data

Labeling data

http://clopinet.com/al


Examples of domains

Examples of domains

  • Chemo-informatics

  • Handwriting and speech recognition

  • Image processing

  • Text processing

  • Marketing

  • Ecology

  • Embryology

http://clopinet.com/al


What is active learning

What is active learning?

http://clopinet.com/al


What is out there

What is out there?

http://clopinet.com/al


Scenarios

Scenarios

Burr Settles. Active Learning Literature Survey.

CDTR 1648, Univ. Wisconsin–Madison. 2009.

http://clopinet.com/al


De novo queries

“De novo” queries

De novo queries implicitly assume interventions on the system under study: not for this challenge

http://clopinet.com/al


Focus on pool based al

Focus on “pool-based” AL

  • Simplest scenario for a challenge.

Training data: labels can be queried

Test data: unknown labels

  • Methods developed for pool-based AL should also be useful for stream-based AL.

http://clopinet.com/al


Example

Example

Accuracy=0.7

Accuracy=0.9

  • Toy 2-class problem, 400 instances Gaussian distributed.

  • Linear logistic regression model trained w. 30 random instances.

  • (c) Linear logistic regression model trained w. 30 actively queried

  • instances using “uncertainty sampling”.

Burr Settles, 2009

http://clopinet.com/al


Learning curve

Learning curve

Burr Settles, 2009

http://clopinet.com/al


Other methods

Other methods

  • Expected model change (greatest gradient if sample were used for training)

  • Query by committee (query the sample subject to largest disagreement)

  • Bayesian active learning (maximize change in revised posterior distribution)

  • Expected error reduction (maximize generalization performance improvement)

  • Information density (ask for examples both informative and representative)

Burr Settles, 2009

http://clopinet.com/al


Datasets

Datasets

http://clopinet.com/al


Data donors

Data donors

This project would not have been possible without generous donations of data:

  • Chemoinformatics -- Charles Bergeron, Kristin Bennett and Curt Breneman (Rensselaer Polytechnic Institute, New York) contributed a dataset, which will be used for final testing.

  • Embryology --Emmanuel Faure, Thierry Savy, Louise Duloquin, Miguel Luengo Oroz, Benoit Lombardot, Camilo Melani, Paul Bourgine, and Nadine Peyriéras (Institut des systèmes complexes, France) contributed the ZEBRA dataset.

  • Handwriting recognition -- Reza Farrahi Moghaddam, Mathias Adankon, Kostyantyn Filonenko, Robert Wisnovsky, and Mohamed Chériet (Ecole de technologie supérieure de Montréal, Quebec) contributed the IBN_SINA dataset.

  • Marketing --Vincent Lemaire, Marc Boullé, Fabrice Clérot, Raphael Féraud, Aurélie Le Cam, and Pascal Gouzien (Orange, France) contributed the ORANGE dataset, previously used in the KDD cup 2009.

    We also reused data made publicly available on the Internet:

  • Chemoinformatics --The National Cancer Institute (USA) for the HIVA dataset.

  • Ecology -- Jock A. Blackard, Denis J. Dean, and Charles W. Anderson (US Forest Service, USA) for the SYLVA dataset (Forest cover type).

  • Text processing -- Tom Mitchell (USA) and Ron Bekkerman (Israel) for the NOVA datset (derived from the Twenty Newsgroups).

http://clopinet.com/al


Development datasets

Development datasets

http://clopinet.com/al


Difficulties

Difficulties

  • Spase data

  • Missing values

  • Unbalanced classes

  • Categorical variables

  • Noisy data

  • Large datasets

http://clopinet.com/al


Final test datasets

Final test datasets

  • Will serve to do the final ranking

  • Will be from the same domains

  • May have different data representations and distributions

  • No feed-back: the results will not be revealed until the end of the challenge

http://clopinet.com/al


Protocol

Protocol

http://clopinet.com/al


Virtual lab

Virtual Lab

Virtual cash

  • Joint work with:

  • Constantin Aliferis, New York University

  • Gregory F. Cooper, Pittsburg University

  • André Elisseeff, Nhumi, Zürich

  • Jean-Philippe Pellet, IBM Zürich

  • Alexander Statnikov, New York University

  • Peter Spirtes, Carnegie Mellon

http://clopinet.com/al


Step by step instructions

Step by step instructions

Download the data. You get 1 labeled example.

  • Predict

  • Sample

  • Submit a query

  • Retrieve the labels

http://clopinet.com/al


Two phases

Two phases

  • Development phase:

    • 6 datasets available

    • Can try as many times as you want

    • Matlab users can run queries on their computers

    • Others can use the labels (provided)

  • Final test phase:

    • 6 new datasets available

    • A single try

    • No feed-back

http://clopinet.com/al


Evaluation

Evaluation

http://clopinet.com/al


Auc score

AUC score

For each set of samples queried, we assess the predictions of the learning machine with the Area under the ROC curve.

http://clopinet.com/al


Area under the learning curve alc

Area under the Learning Curve(ALC)

Linear interpolation. Horizontal extrapolation.

One query

Five queries

Thirteen queries

Lazy: ask for all labels at once

http://clopinet.com/al


Prizes

Prizes

If you win on…

  • 1 dataset: $100

  • 2 datasets: $200

  • 3 datasets: $400

  • 4 datasets: $800

  • 5 datasets: $1600

  • 6 datasets: $3200!

  • Plus travel awards for top ranking students.

http://clopinet.com/al


Schedule

Schedule

http://clopinet.com/al


Conclusion

Conclusion

Try our new challenge, learn, and win!!!!

  • Workshops:

    • AISTATS 2010, Sardinia, May, 2010

    • WCCI 2010 Workshop, Barcelona, July, 2010

    • Travel awards for top ranking students.

  • Proceedings published by JMLR & IEEE.

  • Prizes: P(i)=$100 * 2(n-1)

  • Your problem solved by dozens of research groups:

    • Help us organize the next challenge!

http://clopinet.com/al


  • Login