Quality of Information Research Methodology Colin Legg
SWINE FLU: 50,000 'DEAD IN MONTHS'
SWINE FLU: 50,000 'DEAD IN MONTHS' • NEARLY 50,000 people in Britain could die of swine flu before a vaccine becomes available, it emerged today. • Dr Margaret Chan, head of the World Health Organisation, today rubbished claims by the Government that a reliable flu vaccine would be ready within weeks. • The Government predicts 100,000 new cases-a-day by the end of August. • Chief Medical Officer for England, Sir Liam Donaldson, refused to rule out a death rate of one in 200 - the figure calculated by experts at Imperial College, London yesterday July 15 2009
Mass swine flu coffins warning • MINISTERS have been urged to have thousands of cardboard coffins ready for mass deaths should swine flu take a turn for the worse. • By David Maddox • news.Scotsman.com • Published Date: 17 September 2009
Authority • Who is the author? • What is the source of the information?
Context • A variant of H1N1, Spanish flu, killed 50 million to 100 million people between 1918 and 1919 • What is normal ‘Seasonal flu’ death rate? • “Disease control experts say the death rate from H1N1 is similar to the death rate from seasonal influenza, which kills anywhere between 250,000 and 500,000 people globally each year. “
Case Fatality Ratio • Estimated in Mexico as 0.4% • Multiplier method • Community survey method • Method extrapolating from seasonal influenza mortality • Method extrapolating from a more ‘mature’ epidemic • The four methods produced a wide range of estimates … from 0.0004% to 0.06% Wilson & Baker 2009
Context • H1N1 (swine flu): • has mild symptoms; • low lethality rate; • spreads easily between humans • H5N1 (bird flu): • hardly transmitable between humans; • the lethality rate is over 50%. • What will happen when someone with bird flu catches swine flu as well?
Methodology • Based on real data or model predictions? • What are assumptions of models? • Well established protocols forEstimating infection ratesEstimating cause of deathData handling
Precision • Scope of survey: • Representativeness • Geographic range • Sample size • What level of confidence in the estimates?
Motivation • Are there any financial interests? • Are there any political interests?
Information has huge powerNeed to get your data right Need to be critical of other’s data
The flow of information • Real world • Observation • Notebook • Database • Statistical summary • Interpretation • Publication • Citation • Impact
Observation • Errors of measurement • Noise (sampling error) • Bias
Antarctic ozone hole 3/09/2000 Observation • Errors of measurement • Noise (sampling error)BiasUnrepresentative sample • Machine error • Elite data Use of ‘Indicators’
Sunday Times 10/10/04 “MEN are far less capable than women of retaining self-control when they have been drinking, scientists have found. Their loss of inhibition is more than three times as great as that of women with the same concentration of alcohol in the blood.”
Notebook - database • Copying errors • Selective reporting • Fraud • Fabrication • Falsification (including by omission) • Plagiarism
Statistical summary • Simplification - loss of detail • Statement of precision or confidence • Statistical significance • Consider accuracy
Scientific explanations: • Empirical • controlled conditions of observation • capable of verification by others • Rational • no known false assumptions • no logical errors
Scientific explanations: • Testable • verifiable by direct observation • or make predictions which can be tested • Parsimonious • accept simplest explanation with fewest assumptions
Explanations for the existence of ghosts • Membrane theory • The membrane between this world and the next occasionally wears thin • Stone tape theory • Building materials have a magnetic property that ‘records’ people and events • 3. Deranged attention-seekers theory
Scientific explanations: • Parsimonious • accept simplest explanation with fewest assumptions • General • broad explanatory power in wide range of circumstances
Scientific explanations: • Tentative • willing to accept alternative explanations when assumptions proved false • or more parsimonious explanations when found • Rigorously evaluated • substantial evidence, frequently re-evaluated
Extrapolation • Are your data representative of the population about which you wish to make inferences?
Presentation • ‘How to lie with statistics’ • D Huff (1954, 1991) Penguin Books Ltd
Presentation • ‘How to lie with statistics’ • Present data to best illustrate your point • Remember others are doing the same!
Publication • Typographic errors • Nature and BMJ: • 11.6% & 11.1% errors in stat tests • An error in 38% & 25% of papers • 12% of such errors change P by an order of magnitude • Garcia-Berthou & Alcaraz 2004 • BMC Medical Res Methodology
Funding bias • Funding control • Fashion • Political correctness
Funding bias • Funding control • Fashion • Political correctness • Economic / financial interests
Publication bias Effect of nitrogen fertilizer on concentration of Carbon-based secondary compounds in trees Journal impact rating 0 Leimu & Koricheva 2004 Proc. Roy. Soc. Lond. B Effect size
BBC Panorama 10/10/04 “I believe that it would be impossible to produce an unbiased report when the source of funding came from groups with clearly vested interests.” Professor Jim Mann Committee member
Publication • Publish or Perish • Fabrication, Falsification and Plagiarism • “One in three scientists confesses to having sinned” • Of 3,247 respondents • 1.5% admitted falsification or plagiarism • 15% admitted changing design, methods or results in response to pressure from funding source • 12% overlooked other’s use of flawed data • Nature (2005) 43519: 178-719
Citation process Original data • Kaibab deer Visitor’s estimates Permanent rangers
Citation process Rasmussen 1941 • Kaibab deer Visitor’s estimates Permanent rangers
Citation process Leopold 1943 • Kaibab deer Visitor’s estimates Permanent rangers
Citation process Davis & Golley 1963 • Kaibab deer Visitor’s estimates Permanent rangers
Original data Davis & Golley 1963
Citation process • Kaibab deer • Abstracting journals
Impact • Political decisions • Misuse of data • Social responsibility of science
“The pill carries increased risk of blood clots” • Over 6 month period: • 7-8% rise in rate of conception • 10-13% rise in abortions • = 30,000 extra pregnancies • = 10,000 extra abortions
Conclusion • Information is powerful and valuable,but fragile • Handle with care