Can causal models be evaluated? Isabelle Guyon ClopiNet / ChaLearn

Download Presentation

Can causal models be evaluated? Isabelle Guyon ClopiNet / ChaLearn

Loading in 2 Seconds...

- 106 Views
- Uploaded on
- Presentation posted in: General

Can causal models be evaluated? Isabelle Guyon ClopiNet / ChaLearn

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Can causal models be evaluated?

Isabelle Guyon

ClopiNet / ChaLearn

http://clopinet.com/causalityevents@chalearn.com

- Feature Extraction,
- Foundations and Applications
- I. Guyon, S. Gunn, et al.
- Springer, 2006.
- http://clopinet.com/fextract-book
- 2) Causation and Prediction Challenge
- I. Guyon, C. Aliferis, G. Cooper,
- A. Elisseeff, J.-P. Pellet, P. Spirtes,
- and A. Statnikov, Eds.
- CiML, volume 2, Microtome. 2010.
- http://www.mtome.com/Publications/CiML/ciml.html

Co-founders:

Constantin AliferisAlexander Statnikov

André ElisseeffJean-Philippe Pellet

Gregory F. CooperPeter Spirtes

ChaLearn directors and advisors:

Alexander Statnivov Ioannis Tsamardinos

Richard Scheines Frederick Eberhardt

Florin Popescu

- Motivations
- Quiz
- What we want to do (next challenge)
- What we already set up (virtual lab)
- What we could improve
- Your input…
Note: Experiment = manipulation = action

…your health?

…climate changes?

… the economy?

What affects…

and…

which actions will have beneficial effects?

- Predict the outcome of actions
- What if we ate only raw foods?
- What if we imposed to paint all cars white?
- What if we broke up the Euro?

- Find the best action to get a desired outcome
- Determine treatment (medicine)
- Determine policies (economics)

- Predict counterfactuals
- A guy not wearing his seatbelt died in a car accident. Would he have died had he worn it?

http://data.gov

http://data.uk.gov

http://www.who.int/research/en/

http://www.ncdc.noaa.gov/oa/ncdc.html

http://neurodatabase.org/

http://www.ncbi.nlm.nih.gov/Entrez/

http://www.internationaleconomics.net/data.html

http://www-personal.umich.edu/~mejn/netdata/

http://www.eea.europa.eu/data-and-maps/

Y

Y

X

Y

Y

X

Predict the consequences of actions:

Under “manipulations” by an external agent, only causes are predictive, consequences and confounders are not.

Y

Y

X

If manipulated, a cause influences the outcome…

Y

Y

X

… a consequence does not …

Y

Y

X

… neither does a confounder (consequence of a common cause).

n’

Causal discovery motivations (3) Classical ML helpless

- Special case: stationary or cross-sectional data (no time series).
- Superficially, the problem resembles a classical feature selection problem.

n

X

m

Y

X2

X1

Y

X2

X1

X1 || X2 | Y

Simpson’s paradox

x1

Y

x2

x1

X2

X1

Y

X2

X1

Y

Y

x2

x1

X2 || Y

X2 || Y | X1

baseline

(X2)

health

(Y)

baseline

x2

Y

disease

normal

peak

(X1)

peak

x1

Y

x2

Y

X2

X1

x1

Y

x2

Y

X2

X1

x1

Y

x2

Y

X2

X1

x1

Obstacle 1: Practical

Many statements of the "causality problem"

Obstacle 2: Fundamental

It is very hard to assess solutions

- Experiments are often:
- Costly
- Unethical
- Infeasible

- Non-experimental “observational” data is abundant and costs less.

Experimental design in causal discovery

- Goal: Find variables that strongly influence an outcome
- Method:
- Learn from a “natural” distribution (observational data)
- Predict the consequences of given actions (checked against a test set of “real” experimental data)
- Iteratively refine the model with experiments (using on-line learning from experimental data)

Anxiety

Peer Pressure

Born an

Even Day

QUERIES

Models of systems

Yellow

Fingers

Smoking

Genetics

Allergy

Lung Cancer

Attention

Disorder

Database

Coughing

Fatigue

ANSWERS

Car Accident

Virtual Lab

February 2007: Project starts. Pascal2 funding.

August 2007: Two-year NSF grant.

Dec. 2007: Workbench alive. 1st causality challenge.

Sept. 2008: 2nd causality challenge (Pot luck).

Fall 2009: Virtual lab alive.

Dec. 2009: Active Learning Challenge (Pascal2).

December 2010: Unsupervised and Transfer Learning Challenge (DARPA).

Fall 2012: ExpDeCo (Pascal2)

Planned: CoMSiCo

Setup:

- Several paired datasets (preferably or real data):
- “Natural” distribution
- “Manipulated” distribution

- Problems
- Learn a causal model from the natural distribution
- Assessment 1: test with natural distribution
- Assessment 2: test with manipulated distribution
- Assessment 3: on-line learning from manipulated distribution (sequential design of experiments)

- Largely not relying on “ground truth” this is difficult or impossible to get (in real data)
- Not biased towards particular methods
- Realistic setting as close as possible to actual use
- Statistically significant, not involving "chance“
- Reproducible on other similar data
- Not specific of very particular settings
- No cheating possible
- Capitalize on classical experimental design

Toy datasets

Challenge

datasets

Anxiety

Peer Pressure

Born an

Even Day

Yellow

Fingers

Smoking

Genetics

Allergy

Lung Cancer

Attention

Disorder

Coughing

Fatigue

LUCAS0: natural

Car Accident

Causality assessmentwith manipulations

Anxiety

Peer Pressure

Born an

Even Day

Yellow

Fingers

Smoking

Genetics

Allergy

Lung Cancer

Attention

Disorder

Coughing

Fatigue

Car Accident

Causality assessmentwith manipulations

Anxiety

Peer Pressure

Born an

Even Day

Yellow

Fingers

Smoking

Genetics

Allergy

Lung Cancer

Attention

Disorder

Coughing

Fatigue

Car Accident

Causality assessmentwith manipulations

LUCAS2: manipulated

10

2

5

3

9

4

1

0

6

11

8

7

- We define:
- V=variables of interest
- (Theoretical minimal set
- of predictive variables, e.g.
- MB, direct causes, ...)

- Participants score feature relevance: S=ordered list of features

11

4

1

2

3

- We assess causal relevance with AUC=f(V,S)

P1

P2

P3

PT

Probes

Anxiety

Peer Pressure

Born an

Even Day

Yellow

Fingers

Smoking

Genetics

Allergy

Lung Cancer

Attention

Disorder

LUCAP0: natural

Coughing

Fatigue

Car Accident

Anxiety

Peer Pressure

Born an

Even Day

Yellow

Fingers

Smoking

Genetics

Allergy

Lung Cancer

Attention

Disorder

Coughing

Fatigue

Car Accident

P1

P2

P3

PT

Probes

LUCAP1&2: manipulated

- What we can compute (Fscore):
- Negative class = probes (here, all “non-causes”, all manipulated).
- Positive class = other variables (may include causes and non causes).

- What we want (Rscore):
- Positive class = causes.
- Negative class = non-causes.

- What we get (asymptotically):
Fscore = (NTruePos/NReal) Rscore + 0.5 (NTrueNeg/NReal)

Jianxin Yin: causal

Vladimir Nikulin: non-causal

Simple univariate predictive model, binary target and features, all relevant features correlate perfectly with the target, all irrelevant features randomly drawn. With 98% confidence, abs(feat_weight) < w and Siwixi< v.

ngnumber of “good” (relevant) features

nbnumber of “bad” (irrelevant) features

m number of training examples.

- Leaning curve in terms of number of features revealed
- Without re-training on manipulated data
- With on-line learning with manipulated data

- Give pre-manipulation variable values and the value of the manipulation
- Other metrics: stability, residuals, instrument variables, missing features by design

- We want causal discovery to become “mainstream” data mining
- We believe we need to start with “simple” standard procedures of evaluation
- Our design is close enough to a typical prediction problem, but
- Training on natural distribution
- Test on manipulated distribution

- We want to avoid pitfalls of previous challenge designs:
- Reveal only pre-manipulated variable values
- Reveal variables progressively “on demand”