News from the bioconductor project genetic interactions multiple testing and independent filtering
1 / 51

News from the Bioconductor project Genetic interactions Multiple testing and independent filtering - PowerPoint PPT Presentation

  • Uploaded on

News from the Bioconductor project Genetic interactions Multiple testing and independent filtering. Praha, 26 Aug 2009 Wolfgang Huber EMBL. An international open source and open development software project for the analysis of genomic data

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' News from the Bioconductor project Genetic interactions Multiple testing and independent filtering' - shalin

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
News from the bioconductor project genetic interactions multiple testing and independent filtering
News from the Bioconductor projectGenetic interactionsMultiple testing and independent filtering

Praha, 26 Aug 2009

Wolfgang Huber


An international open sourceand open development software project for the analysis of genomic data

Use the statistical environment and language R as the integrating middleware

Design principles: rapid development, code re-use

Six-monthly release cycle; release 1.0 in March 2003 (15 packages), …, release 2.4 in April 2009 (320 packages)‏

  • Goals

  • Provide access to powerful statistical and graphical methods for the analysis of genomic data

  • Facilitate the integration of biological metadata (e.g. EntrezGene, BioMarts, PubMed) in the analysis of experimental data

  • Promote the development of accessible, extensible, transparent and well-documented software

  • Promote reproducible research

  • Provide training in computational and statistical methods

Good scientific software is like a good scientific publication
Good scientific software is like a good scientific publication


Subject to peer-review

Easy to access and use by others

Builds on the work of others

Others can build their work on top of it

  • 320 packages in release 2.4, April 09 (60 more than last year)

  • Packages represent the work of 210 different researchers

  • 76 Pubmed citations between January 2008 and May 2009

  • Original Bioconductor paper has over 1450 citations

  • Website sees over 14,000 unique visitors per month

  • Biobase package was downloaded to 34,354 unique IP addresses in the last year

  • Mailing lists have nearly 3000 subscribers, ~450 posts per month

  • New list dedicated to short read sequence analysis

  • In 2008, 8 Bioconductor courses; all at or near capacity

  • 80 participants attended annual conference, BioC2009

Bioconductor year)Short Course: Brixen, South Tyrol

annually in June since 2004, 1 week

Bioconductor Conference: Seattle, WA, annually late July

Developer Meeting: Dec 09 in Manchester UK

Many further short courses & developer meetings: see!

How to get started
How to get started year)

Go to and follow installation instructions (Linux, Windows, Mac)

Read the book

Subscribe to the mailing list

Answer questions

Contribute your own package!

Recent developments next generation sequencing
Recent developments: year) next-generation sequencing

ShortRead package

data management (raw reads, alignments to reference genome e.g. from Bowtie, SAM-tools)

data quality assessment


efficient string and alignment algorithms let you experiment with your own ideas


efficient working with interval-type data and annotation along chromosomes

Recent developments next generation sequencing1
Recent developments: year) next-generation sequencing

BSgenome data packages

genome sequences of many model organisms


view your data with a genome browser


2D embedding (Hilbert-curve) plots of along-genome data


identify enriched regions


make your own, custom along-chromosome plots

Hilbert plots of chromosome 10 year)



3-colour Hilbert plot year)







HilbertVis package

Recent developments preprocessing and data quality report generation
Recent developments: year)preprocessing and data quality report generation


short read data


expression microarrays


Flow cytometry


Recent developments high content high throughput assays
Recent developments: year) high content / high throughput assays


RNAi and compound screening with cell-based assays

flow*gating and normalisation for flow cytometry data


segmentation of 2D images of cells, feature extractionimageHTS (coming soon)

complete data analysis workflow for large-scale microscopy imaging based screens

Complex traits
Complex traits year)

Do not follow Mendelian inheritance and result from multiple alleles

Responsible alleles contribute different amounts to phenotype

Alleles may be present in only a fraction of all individuals with the phenotype

The genetic basis of complex traits
The genetic basis of complex traits year)

Currently ~400 variants that contribute to common traits and diseases are known

Even when dozens of genes have been linked to a trait, both the individual and the cumulative effects are disappointingly small (<5%)

Limitations to current genome-wide association studies:

Common SNPs miss rare variants with potentially huge effects

Common variants with low penetrance also missed

Many structural variants go undetected because they do not alter SNP sequences

Epistasis, where the effect of one variant cannot be found without knowing the other, confounds identification

Picture emerges that complex traits are conditioned by many common variants with different effect sizes and frequencies

Genetic architecture of complex traits: Large year)phenotypic effects and pervasive epistasisSinger et al. Science 2004, Shao et al. PNAS 2008 (J. Nadeau lab)



differ for many physiological, morphological, behavioral, immunological, and oncological traits, including models of birth defects and adult diseases in humans

Css panels
CSS Panels year)

Chromosome substitution strains (CSS) panel:

Strain CSS-i carries both copies of chromosome i from donor strain, but all other chromosomes from the host strain are intact and homozygous.

Creation of C57BL/6J - A/J panel required more than 17,000 mice and took about 7 years. (With up-to-date selective breeding and genotyping, now estimated to be feasible in 4 years.)





Phenotyping of 90 different complex traits
Phenotyping of 90 different complex traits year)

41 traits significantly different between parentals

Per trait, there are on average 8 significant CSSs

Average phenotypic effect per chromosome: 76% of the total

Interactions year)

Additive model for quantitative trait Y

General model with interactions

Xi : genotype at locus i

bij : interaction effect of loci i, j

bi : main effect of locus i

Types of interactions

viability year)
















Types of interactions

positive (synergistic) interaction

negative (buffering)‏




negative (suppressing)






Combinatorial RNAi to systematically map all pairwise interactionsT. Horn, T. Sandmann, M. Boutros (DKFZ); E. Axelsson (EBI)



384-well plate

384-well plate

>3500 PP x PP (quadruplicates)

PP x positive controls

PP x negative controls

PP alone

Arrays of RNAi reagents

Estimating genetic interactions interactions

Data: log (cell viability)

viability hits

Pij = Wi + Wj + Wij

obtain estimates for Wi, Wj, Wij by minimizing

single effects

synergistic interactions

antagonistic interactions


Related phosphatases share interaction profiles

cluster of lipid phosphatases interactions

(CG11437, CG11438, wun)

interact with JNK signaling

Related phosphatases share interaction profiles




Example rescue from diap1
Example: rescue from DIAP1 interactions

DIAP, Drosophila Inhibitor of Apoptosis

co-depletion with P71 shows rescue phenotype

reproducible in different cell lines






negative (suppressing)


Protein networks based on synthetic genetic interactions
Protein networks based on synthetic genetic interactions interactions

  • Limited number of PP with many interactions, e.g.

    • puc

    • PP1- a96

  • Expansion to larger network sizes, more complex phenotypes

  • Application to RNAi-drug interactions

Next Steps interactions

Microscopy Imaging - multivariate phenotypes: not just overall cell number, but morphological parameters of each invidual cell (cytokinesis defects, differentiation, apoptosis)

Scale up to larger matrices (1000 x 1000 is feasible)

Not just overall knock-down, but allele-specific perturbations in diploid backgrounds

Aim: multivariate phenotype-genotype models with predictive value for phenotype engineering and rational drug therapy

Independent filters and multiple testing

Independent filters and interactionsmultiple testing

Wolfgang Huber (EMBL/EBI)

Richard Bourgon (EMBL/EBI)

Robert Gentleman (FHCRC)

Multiple testing
Multiple testing interactions

  • Many data analysis approaches in genomics rely on item-by-item (i.e. multiple) testing:

  • Microarray expression profiles of “normal” vs “perturbed” samples: gene-by-gene

  • ChIP-chip: locus-by-locus

  • RNAi and chemical compound screens

  • Genome-wide association studies: marker-by-marker

  • QTL analysis: marker-by-marker and trait-by-trait

Multiple testing1
Multiple testing interactions

  • Classical hypothesis test:

  • null hypothesis H0, alternative H1

  • test statistic X  t(X) 

  • a = P( t(X) Grej | H0) type I error (false positive)

  • b = P( t(X) Grej | H1) type II error (false negative)

  • When n tests are performed, what is the extent of type I errors, and how can it be controlled?

  • E.g.: 20,000 tests at a=0.05, all with H0 true: expect 1,000 false positives

Experiment wide type i error rates
Experiment-wide type I error rates interactions

Family-wise error rate: P(V > 0), the probability of one or more false positives. For large m0, this is difficult to keep small.

False discovery rate: E[ V / max{R,1} ], the expected fraction of false positives among all discoveries.


p-values: a mixture interactions




Example differential expression testing
Example: differential expression testing interactions

Acute lymphocytic leukemia (ALL) data, Chiaretti et al., Clinical Cancer Research 11:7209, 2005

Immunophenotypic analysis of cell surface markers identified

T-cell derivation in 33,

B-cell derivation in 95 samples

Affymetrix HG-U95Av2 3’ transcript detection arrays with ~13,000 probe sets

Chiaretti et al. selected probesets with “sufficient levels of expression and variation across groups” and among these identified 792 differentially expressed genes.

Clustered expression data for all 128 subjects, and a subset of 475 genes showing evidence of differential expression between groups

Independent filtering
Independent filtering interactions

From the set of 13,000 probesets,

first filter out those that seem to report negligible signal (say, 40%),

then formally test for differential expression on the rest.

Conditions under which we expect negligible signal :

Target gene is absent in both samples. (Probes will still report noise and cross-hybridization.)

Probe set fails to detect the target.

Literature: von Heydebreck et al. (2004)McClintick and Edenberg (BMC Bioinf. 2006) and references thereinHackstadt and Hess (BMC Bioinf. 2009)

Many others.


Increased detection rates
Increased detection rates interactions

Stage 1 filter: compute variance, across samples, for each probeset, and remove the fraction θ that are smallest

Stage 2: standard two-sample t-test

ALL data

Increased detection rate implies increased power interactionsonly if we are still controlling type I errors at the nominal level.

Increased power?

  • Concerns:

  • Have we thrown away good genes?

  • Use a data-driven criterion in stage 1, but do type I error consideration only on number of genes in stage 2

  • Informal justification:

  • Filter does not use covariate information (T-/B-cell type)

ALL data


Non specific filtering
Non-specific filtering? interactions

  • An informal explanation has been that the filtering “does not use any information from the class labels”. However, this is not enough, as these examples show:

  • Unsupervised clustering of the samples into two groups, filter by t-statistic for these groups, then test by t-statistic for actual groups. Asymptotically, and for certain data,

  • stage 1 statistic  stage 2 statistic

  • 2. Certain null distributions of the data that are not rotation symmetric (but iid).Then,

  • Lt ≠ Lt | s

Result independence of stage 1 and stage 2 statistics under the null hypothesis
Result: independence of stage 1 and stage 2 statistics under the null hypothesis

For genes for which the null hypothesis is true (X1 ,..., Xn exchangeable), f and g are statistically independent in both of the following cases:

• Normally distributed data:

f (stage 1): overall variance (or mean)

g (stage 2): the standard two-sample t-statistic, or any test statistic which is scale and location invariant.

• Non-parametrically:

f: any function that does not depend on the order of the arguments. E.g. overall variance, IQR.

g: the Wilcoxon rank sum test statistic.

Both can be extended to the multi-class context: ANOVA and Kruskal-Wallis.


Derivation the null hypothesis

Non-parametric case:

Straightforward decomposition of the joint probability into product of probabilities using the assumptions.

Normal case:

Use the spherical symmetry of the joint distribution, p-dimensional N(0, 1s2), and of the overall variance; and the scale and location invariance of t.

This case is also implied by Basu's theorem

(V complete sufficient for family of probability measures P, T ancillary  T, V independent)

Type i error control requires
Type I error control requires the null hypothesis

  • 1. Correct specification of the the marginal distribution of the test statistic for the true nulls.

  • 2. A dependence structure which is appropriate for the multiple testing method being used.

more subtle

How multiple testing procedures deal with dependence
How multiple testing procedures deal with dependence the null hypothesis

  • 1. Methods that work on the p-values only and allow general dependence structure: Bonferroni, Bonferroni-Holm (FWER), Benjamini-Yekutieli (FDR)

  • 2. Those that work on the data matrix itself, and use permutations to estimate null distributions of relevant quantities (using the empirical correlation structure): Westfall-Young (FWER)

  • 3. Those that work on the p-values only, and require dependence-related assumptions: Benjamini-Hochberg (FDR), q-value (FDR)

q good? (power)

Diagnostics good? (power)

odds ratio = 4.4

abs. value of t-statistic (stage 2)

ALL data

overall standard deviation (stage 1)

Results summary
Results summary good? (power)

  • There are cases in which "filtering" leads to incorrect type-I error control.

  • In other cases, the stage-one (filter) and stage-two (differential expression) statistics are marginally independent:

    • (Normal distributed data): overall variance or mean, followed by t-test

    • Any permutation invariant statistic, followed by Wilcoxon rank sum test

  • Marginal independence is sufficient to maintain control of FWER at nominal level.

  • Marginal independence does not preclude changes to correlation structure in filtered data: control of FDR not guaranteed; this is not likely a problem in practice.

Conclusion good? (power)

  • Correct use of this two-stage approach can substantially increase power at same type I error.

  • Why does it work?

  • The filtering step is an (informal) way to bring in additional knowledge about the data (a „model refinement“)

EMBL good? (power)

Premier lab for biological research in Europe, with five sites, in Heidelberg, Cambridge (UK), Grenoble (F), Rome and Hamburg.

Cell biology, Biophysics, Developmental Biology, Structural Biology, Genome Biology, Computational Biology


Simon Anders good? (power)

Elin Axelsson

Ligia Bras

Richard Bourgon

Bernd Fischer

Audrey Kauffmann

Gregoire Pau

Thank you

Robert Gentleman, F. Hahne, M. Morgan (FHCRC)

Lars Steinmetz, J. Gagneur, Z. Xu, W. Wei (EMBL)

Michael Boutros, F. Fuchs, D. Ingelfinger, T. Horn, T. Sandmann (DKFZ)

Steffen Durinck (Illumina)

All contributors to the R and Bioconductor projects

Derivation non parametric case
Derivation (non-parametric case) good? (power)

  • exchangeability

  • A, B: measureable sets

  • f: stage 1, g: stage 2

Parametric case: seeRichard Bourgon‘sposter

  • f's permutation invariance

  • distribution of g generated by permutations