Download
analysis of the full ewing ews fli screen n.
Skip this Video
Loading SlideShow in 5 Seconds..
Analysis of the Full Ewing EWS/FLI Screen PowerPoint Presentation
Download Presentation
Analysis of the Full Ewing EWS/FLI Screen

Analysis of the Full Ewing EWS/FLI Screen

120 Views Download Presentation
Download Presentation

Analysis of the Full Ewing EWS/FLI Screen

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. The Broad Institute of MIT and Harvard Analysis of the Full Ewing EWS/FLI Screen Ken Ross 10/22/10

  2. Outline • Review of analysis pipeline • Analysis of Ewing EWS/FLI screen • Screen overview • June 2010 screen issues and bad plates • Bad plates selected to be repeated • Plates with known technical problems • Plates with low fraction of genes changing in correct direction • Plates with poor summed score and weighted summed score z-factors • Plates with high hit rates • Combined screen • Good plates from June 2010 screen • Plates repeated from June screen • Pilot screen data • Viability prediction • Hits • Summary and conclusions

  3. Zhang et al. 1999 Z-Factor: • Z-factor=1 would be ideal • Z-factor > 0 are good – more than 3 standard deviations separates control means • Typically we see z-factor > 0.5 for EWS/FLI plates Screen Analysis Methodology • Data processing includes: • Filtering • Well scaling by forming ratio to reference genes • Scaling and normalization • Compounds scored based upon a combination of methods: • Summed score • Weighted summed score • Naïve Bayes • KNN classifier • SVM classifier • Trading-off options for analysis parameters was partially based upon maximizing the Z-factor

  4. Naïve Bayes Classifier • Based upon the Bayes rule and “naively” assumes feature independence Pr[h|E] = (Pr[E1|h] x Pr[E2|h] x … x Pr[EN|h] x Pr[h])/Pr[E] • where Ei is the evidence for the hypothesis (in this case, gene ratios as evidence for the cells transforming) • Probabilities for continuous variables (like gene expression ratios) are modeled as independent Gaussian or kernel distributions • Naïve Bayes works even when the independence assumption does not hold

  5. project samples in gene space classblack gene 2 classorange gene 1 K-nn classifier example: K=5, 2 genes, 2 classes

  6. ? K-nn classifier example: K=5, 2 genes, 2 classes project unknown sample classblack gene 2 classorange gene 1

  7. Distance measures: • Euclidean distance • 1-Pearson correlation • KL divergence • … ? K-nn classifier example: K=5, 2 genes, 2 classes "consult" 5 closest neighbors: -3 black - 2 orange classblack gene 2 classorange gene 1

  8. Distance measures: • Euclidean distance • 1-Pearson correlation • KL divergence • … K-nn classifier example: K=5, 2 genes, 2 classes "consult" 5 closest neighbors: -3 black - 2 orange classblack gene 2 classorange gene 1

  9. Support Vector Machine (SVM) Prediction • A SVM maps input vectors to a higher dimensional space where a maximal separating hyperplane is constructed • Parallel hyperplanes are constructed on each side of the hyperplane that separates the data • The separating hyperplane is the hyperplane that maximizes the distance between the parallel hyperplanes • Assumes that a larger margin or distance between the parallel hyperplanes results in a classifier with a better generalization error

  10. Ewing EWS/FLI Screen Overview • 36 chemical library plates (31 in June 2010 screen and 5 in pilot screen) • Screened in duplicate • DOS libraries (25 plates – 8000 compounds) • Natural products (4 plates – 1280 compounds) • Commercial compounds (2 plates – 640 compounds) • Bioactive compounds (6 plates – 1920 compounds) • Positive Control: EWS/FLI knockdown (32 per plate) • Negative Control: DMSO (32 per plate) • LMA plates generated and detected by the GAP • 138 gene signature for readout (134 pilot) • 6 reference genes • 89 genes up-regulated by EWS/FLI knockdown • 49 genes down-regulated by EWS/FLI knockdown (45 pilot)

  11. June 2010 Screen Quality Overview • 8 plates were obviously bad and needed to be redone • 6 had bad PCR • 1 flipped plate (poor performance when un-flipped, possibly flipped back and forth during wash) • 1 plate with incorrect detector program • Remaining plates were processed in many batches with obvious batch effects • Problems were evident in some plates with: • Low fraction of genes changing in expected direction • Plates with fraction of good genes changing in the expected direction < 0.7 were considered bad – 6 plates • Poor z-factors for summed score • Summed score z-factors < -0.5 were considered bad (good genes) – 5 plates • Poor z-factors for weighted summed score • Weighted summed score z-factors < 0.2 were considered bad – 6 plates • Calculated platewise weights with good genes because of plate-to-plate variation • Large group of wells without beads / overall low bead count • Excessive number of hits • Plate considered to have high number of hits if SS hits > 50, WSS hits > 50, Naïve Bayes hits > 60, or KNN hits >10

  12. Summary of Problematic Plates from June 2010 • 24 plates total • 2 high hit count plates are replicates of same chemical plate – might be real • 2 other chemical plates have problems with both replicates • 20 repeated in September 2010 • One low bead count plate (BR00022351) was ok after redetection • One with marginal WSS z-factor (0.1) and marginally low bead counts kept (BR00021940) • Two with moderately high hits rates in only SS and Naïve Bayes in both replicates kept (BR00021953 and BR00022368)

  13. EWS/FLI Screen Analysis Approach • Screen consists of 42 plates from June 2010, 20 plates repeated in September 2010, and 10 plates from pilot screen • Each plate analyzed separately • Positive Control: EWS/FLI knockdown (32 per plate except pilot has 16) • Negative Control: DMSO (32 per plate except pilot has 16) • Filtering: • Reference gene: GAPDH median EF3-2 minus 4 median absolute deviations (with a minimum level of 6000) • Bead count: more than 10 probes with 6 or fewer beads • Well normalization • Marker genes ratioed to mean of 3 reference genes (ACTB, HINT1, and TUBB) • Each plate analyzed twice: • All genes • Good Genes – Genes are considered ‘good’ if they change in the expected direction and have z-factors > -30 • Five methods are used to evaluate hits on each plate and then hit lists are combined together • Summed Score • Weighted Summed Score • Naïve Bayes • KNN • SVM

  14. Summed Scores (All Genes) • Batch effects are obvious here • Recent batch looks much better Sept. 2010 Plates Pilot Plates June 2010 Plates DMSO EF3-2 Knockdown Compounds Luciferase

  15. Summed Scores (Good Genes) Sept. 2010 Plates • Batch effects are obvious here • Different good genes on each plate exaggerates plate-to-plate differences Pilot Plates June 2010 Plates DMSO EF3-2 Knockdown Compounds Luciferase

  16. Z-Score of Summed Scores (All Genes) • Z-score helps make score comparable among plates • Recent batch still looks better Sept. 2010 Plates Pilot Plates June 2010 Plates DMSO EF3-2 Knockdown Compounds Luciferase

  17. Z-Score of Summed Scores (Good Genes) • Z-score helps make score comparable among plates • Even with the different size signatures scores seem comparable Sept. 2010 Plates Pilot Plates June 2010 Plates DMSO EF3-2 Knockdown Compounds Luciferase

  18. Heatmap for EWS/FLI Screen Plate Means Up Genes Down Genes DMSO EF3-2 Knockdown Compounds Luciferase

  19. EWS/FLI Screen Plate Z-Factors (All Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • No scaling • Ratios with the 3 best reference genes • All genes • Mean Z-factors: • Summed score mean z-factor = 0.36 • Weighted summed score mean z-factor = 0.45 • All z-factors are better than thresholds set after June 2010 screen

  20. EWS/FLI Screen Plate Z-Factors (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • No scaling • Ratios with the 3 best reference genes • Good genes • Mean Z-factors: • Summed score mean z-factor = 0.35 • Weighted summed score mean z-factor = 0.51 • All z-factors are better than thresholds set after June 2010 screen

  21. Fraction of Genes Changing in Expected Direction • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • No scaling • Ratios with the 3 best reference genes • Good genes • Mean fraction of genes changing in expected direction = 0.74 • All plates have 65% or more of all genes changing in expected direction

  22. Fraction of Genes Changing in Expected Direction • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • No scaling • Ratios with the 3 best reference genes • Good genes • Mean fraction of genes changing in expected direction: • Up genes = 0.84 • Down genes = 0.56 • All genes = 0.74 • The balance of up and down genes changing varies considerably

  23. Histogram of Z-Score of Summed Scores (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • No scaling • Ratios with the 3 best reference genes • Good genes • Summed score z-score normalized after calculation so plate data can be compared

  24. Histogram of Z-Score of Weighted Summed Scores (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • No scaling • Ratios with the 3 best reference genes • Good genes • Plate specific weights • Weighted summed score z-score normalized after calculation so plate data can be compared

  25. Hit Distribution (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • Ratios with the 3 best reference genes • Good genes • Plate specific weights • 5 methods for calling hits • Summed score probability > 0.5 • Weighted summed score probability > 0.5 • Naïve Bayes > 0.5 • KNN • SVM

  26. Summed Scores Hit Distribution (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • No scaling • Ratios with the 3 best reference genes • Good genes • Not a linear relationship between score and probability but the best probabilities have the highest scores

  27. Weighted Summed Scores Hit Distribution (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • No scaling • Ratios with the 3 best reference genes • Good genes • Plate specific weights • Not a linear relationship between score and probability but the best probabilities have the highest scores

  28. Hit Distribution with Scores (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • Ratios with the 3 best reference genes • Good genes • Plate specific weights • 5 methods for calling hits • Summed score probability > 0.5 • Summed score z-score > 3 • Weighted summed score probability > 0.5 • Weighted summed score z-score > 3 • Naïve Bayes > 0.5 • KNN • SVM

  29. Hit Distribution Versus Plate (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • Ratios with the 3 best reference genes • Good genes • Plate specific weights • 7 methods for calling hits • Summed score probability > 0.5 • Summed score z-score > 3 • Weighted summed score probability > 0.5 • Weighted summed score z-score > 3 • Naïve Bayes > 0.5 • KNN • SVM

  30. Hit Distribution Versus Plate Pairs (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • Ratios with the 3 best reference genes • Good genes • Plate specific weights • 7 methods for calling hits • Summed score probability > 0.5 • Summed score z-score > 3 • Weighted summed score probability > 0.5 • Weighted summed score z-score > 3 • Naïve Bayes > 0.5 • KNN • SVM

  31. Paired Plate Hit Distributions (Good Genes) Summed Score Probability > 0.5 Hits Naïve Bayes Probability > 0.5 Hits Z-Score of Summed Score > 3 Hits KNN Hits Weighted Summed Score Probability > 0.5 Hits SVM Hits Z-Score of Weighted Summed Score > 3 Hits

  32. Viability Prediction in Ewing Data • Viability prediction model developed with data from 3 development plates and pilot screen: • Cytotoxic compound plate • Compound plate from Kung lab with compounds predicted active in Ewing sarcoma • Phenothiazine compound plate • EWS/FLI pilot screen data (5 chemical plates in duplicate from ChemBiology) • Viability prediction with KNN • Sample relative viability defines cells in well as either alive (>=0.2) or dead (<0.2) • Development data split into train and test with 25% of combined data from 3 development plates and pilot screen for training and remaining for testing • Data Median Range Scaled together and each sample normalized (subtract median and divide by median absolute deviation)

  33. Classifiers for Viability Prediction25% of all Data for Training – KNN – Normalized 25% Combined Training – LOOCV Results • Ewing development plate with 25% of data from 3 development plates and pilot screen used to train a viability model • Used day 2 viability relative to DMSO mean of day 2 viability relative to day 0 ratio • Sample relative viability defines cells in well as either alive (>=0.2) or dead (<0.2) • Median range scaled data • Samples normalized by subtracting median and dividing by median absolute deviation • KNN classifier • K=3 • Cosine distance • Distance weighting • 10 features selected by signal-to-noise • Performance with normalization slightly better Held-Out 75% Combined Testing

  34. Applying Viability Prediction to EWS/FLI Screen • Ewing development plate with 25% of data from 3 development plates and pilot screen used to train a viability model • Used day 2 viability relative to DMSO mean of day 2 viability relative to day 0 ratio • Sample relative viability defines cells in well as either alive (>=0.2) or dead (<0.2) • Median range scaled data • Samples normalized by subtracting median and dividing by median absolute deviation • KNN classifier • K=3 • Cosine distance • Distance weighting • 10 features selected by signal-to-noise • Seems to have some success • Some known toxic compounds are predicted dead • Probably many of the 389 of 23040 compound wells filtered would also be predicted dead

  35. Hit Distribution with Viability (Good Genes) • 72 plates (42 from June 2010, 20 from September 2010, and 10 from pilot screen) • Each plate analyzed separately • Ratios with the 3 best reference genes • Good genes • KNN model used to predict viability • Trained on pilot screen and development plates • 5 methods for calling hits • Summed Score Probability • Weighted Summed Score Probability • Naïve Bayes • KNN • SVM

  36. Viability and Subsignature PredictionConclusions / Future Work • Viability prediction seems to be working reasonably well • A reasonable number of wells that made it through reference gene filtering are being classified as “dead” • Many of the hits appear to be in “dead” wells • Evaluate with secondary screen where there will be viability measurements • More work with viability prediction • Need to further explore methods for working across different batches of data • How much training data is needed? Would more training data improve the models? • What kind of training samples are needed? Can we use several standard test plates? • Further evaluation of data scaling, use of log expression ratio, and other model types • Try other types of classifiers, e.g., SVM and Naïve Bayes • Other work with subsignatures in GE-HTS data

  37. Conclusions / Future Work • Quality of screen data is not ideal but workable • Repeat plates and pilot screen plates have best quality • Key to salvaging poor data is ability to follow-up a sufficiently large number of hits • Analysis methods needed to accommodate data obtained from many batches and to be robust to batch variations • Analyzed plates separately and then collapsed results • Used plate specific models for hit selection • Each plate used its own set of ‘good’ genes (plate dependant) • Fortunately 32 positive and 32 negative controls on each plate • SVM seems to suffer the most with plate-by-plate analysis because wide separation between controls allows divergent models • Overlap of hits between replicates suggests that the batch effect problems have been at least partially dealt with • Can the predicted viability be used to avoid ‘hits’ that just kill the cells?