Understanding Risk

Understanding Risk An adventure in the perception of news Rebecca Goldin, Ph.D. Director of Research, STATS Professor of Mathematics, GMU January 17, 2013 National Press Foundation

Statistical Assessment Service www.stats.org Jon Entine, Senior Fellow Cynthia Merrick, Intern Rebecca Goldin, Director of Research Trevor Butterworth, Editor

Statistical Concepts in Writing about Risk • Mean, median, mode • Standard deviation • Confidence intervals • Orders of magnitude • Confounding factors • Percentages • Absolute vs. relative risk • Scientific methods • Causation versus correlation

Absolute versus relative risk • Absolute risk is the risk you actually undergo. Women who take the birth control pill have an absolute risk of venous thrombosis (blood clot) of about 1 in 10,000 per year. The absolute risk of women who do not take the pill is 1 in 15,000 per year. • Relative risk is a risk compared to another group. Women who take the birth control pill have a 50% increased risk of venous thrombosis, compared to women who don’t take the pill.

The absolute risk for DVT was 15 per 100,000 for 2nd generation birth control pills The absolute risk for DVT was 30 per 100,000 for 3rd generation birth control pills. The media blitz led to many women not taking their medications (rather than immediately replacing them) and an increase in unwanted pregnancies The abortion rate went up 9% from 1995 to 1996. The absolute rate of DVT for pregnant women is 80 per 100,000. Relative risk representations have consequence • In 1995, Committee on Safety of Medicine in UK concluded that the 3rd generation birth control pill was riskier than previous versions. • Some press reported a 100 percent increase in risk in Deep Vein Thrombosis (blood clots); others reported “twice the risk”. • These are relative risks.

Kids dying from the flu Detroit Free Press (Jan 16, 2013): “Risk to all ages: About 100 children die of flu each year” Story has many specific examples of children dying, with parents describing the tragedies Prevalence of flu vaccine discussed (About 40% of children are vaccinated) Effectiveness of flu discussed (60% of people who would have gotten it do not). Most deaths among those not vaccinated 24,000 per year die of all ages in the U.S. No mention of absolute risk: there are about 74 million children in the United States. The risk of death by flu is 1 in 740,000 per year, or .00014 percent.

Comparisons are powerful for risk Cars Guns ~250,000,000 registered cars in the United States (Department of Transportation) ~32,000 car crash fatalities (National Highway Traffic Safety Administration, 2010) 12.3 road fatalities per 100,000 people, 1.5-3 times as many as in Europe (8.7 in Italy, 6.9 in France, 4.5 in Germany) ~130,000 federally licensed firearms dealers in the United States (Bureau of Alcohol, Tobacco, Firearms and Explosives); ~150,000 gas stations, ~14,000 McDonald’s 270,000,000 guns in circulation (survey, 2007)) ~30,000 people die each year including 8,500 by murder (2011, FBI), 19,000 by suicide (CDC, 2009), 1,000 by accident Rate of death by firearm is about 3-8 times as much as in European countries.

Causation or Correlation

It’s easy to be fooled • Height correlates with reading skills in children under 10. • Income correlates with success in college. • Ratio of finger lengths correlates with aggression. • Facebook correlates with poor grades. • Facebook correlates with good grades. • Doing heroin correlates with doing marijuana. • Higher taxes correlate with high annual growth, and are inversely correlated with poverty rates. • Alcoholism correlates with less gray matter in the prefrontal cortex.

“IVF pregnancies may increase risk of blood clots, blocked arteries” Fox News, Jan 16, 2013 • In vitro fertilization “may come with a slight associated risk: blood clots and blockages.” • “previous studies have found IVF to be just as safe as normal pregnancies, but [the authors of the new study] weren’t necessarily convinced” • Study Design: About 24,000 women who had undergone IVF were compared to about 115,000 women who had normal pregnancies. Each had average age of 33.

“IVF pregnancies may increase risk of blood clots, blocked arteries” Fox News, Jan 16, 2013 • Those who went IVF were more likely to have blood clots. 4.2 out of 1000 had Venous Thrombembolism among IVF women, compared to 2.5 out of 1000 for other women. • .08 percent of IVF women had a blocked artery, while only .05 percent of women with normal pregnancies did. • But is it the IVF that increased the risk, or is IVF reflective of an increase level of risk? Women in each group do not have the same health profile. • Non-IVF contributors to these different numbers: different levels of fertility, access to medical care and diagnosis, hormonal treatments not directly related to IVF • IVF pregnancies carry more risk of blood clots and blocked arteries, but evidence IVF increases the risk is weak based on this report alone.

fMRI studies… a case study • fMRIs are large magnetsmeasuring oxygen levels inblood • People can engage inactivities inside the machine • Patterns of blood flow arethought to reflect patternsof brain activity (more onthat in a bit). • Typical studies: assume that observed patterns only occur when the tested behavior occurs. • Typical studies: assume that observed patterns are caused by the tested behavior.

fMRI studies… a case study • Lying can be determined by patterns of fMRI scans. But perhaps stress or anxietycan lead to the same patterns • Violent video gaming leadsto violent brain patterns But perhaps any competitive play, including non-violent non-video games has similar brain patterns. Plus, no indication of actual violence. • Math anxiety triggers activity in the pain center of the brain. But no pain experienced by subject with math anxiety. Perhaps anxiety, not mathematics, correlated with measured brain activity.

Jumping from Correlation to Cause • You don’t always have to know why it may not be causal. Be wary of any claims of causality. • Some common reasons that a correlation could look causal when it’s not include: not adjusting for confounders, misunderstanding the mechanism, having an unknown confounder. • A causal relationship might be reasonable to suspect when the statistics are • Overwhelming • Observed in many different contexts • Repeated tests show the same effect, on large numbers of people • Double blind case-control studies.

Causation vs. correlation is not the only thing to worry about in medical research

The roll of randomness • Given a hug urn of balls – 30% of the balls are white, and the rest are other colors. • Each of 100 people pick10 balls, write down theircolors, then return theballs to the urn. • Some people will have 3 white balls, but otherswill have greater or fewer.

Number of White Balls is Random • Suppose that “white” represents something random, and bad, like the number of cancers per 100 people in a town. • If one town gets 7 cancers per 100 (double the expected number), wouldn’t you think there’s a reason? Our statistics suggest maybe not. But we are often convinced of causality. • About 27% chance you will get 3 white balls; it’s much more likely you’ll get some other number • About 1% chance of getting 7 white balls

Randomness has structure You can predict how likely the data is, if you assume the probability of white is 30%.

On p-values (how likely am I to see the data I see, if the data are random?) • Suppose you are flipping a coin many times, and you think this coin is biased, because you aren’t getting close to ½ heads and ½ tails. How can you quantify your suspicion?

On p-values (how likely am I to see the data I see, if the data are random?) • The p-value is a measurement based on the data you have seen: it answers the question: “if the coin were fair, how likely would I be to see the data I am seeing?” In other words, if you had a fair coin, is it reasonable to see the proportion of heads/tails, or is it very unlikely to see that? • If you flip 1000 times, and you get 520 heads, there is just under a 10% chance of getting this many heads (or more). In contrast, if you had 550 heads in 1000 flips, the chance of this happening randomly is only about .1%., i.e. very unlikely if the coin were fair. • The biomedical community generally accepts p=.05 (5%) as a standard for when you can reject the notion that the coin was fair.

Multiple Testing, or How to Guarantee Results • Once you have a standard, like p<.05, you have ways of gaming the system. There is always a small chance, something less than 5%, that you will see something that looks suspicious when it really isn’t. Sometimes your coin will favor heads by a suspicious amount, the coin really is fair. • The more hypotheses you check, the more likely this it to happen. • And once you find something suspicious, you can write a scientific article about that.

What happens in the lab:Experiments Galore...

What the rest of the world sees

Metaphors for Bad Statistical Methods • Drunk looking for his keys under the lamp post…

Metaphors for Bad Statistical Methods • Texas Sharpshooter Fallacy…

“Depression linked to increased stroke mortality” – Heartwire • Based on data from NHANES I Epidemiologic Follow-up Study (National Health and Nutrition Examination Survey) • But if you just comb the data, you are likely to find something juicy. This doesn’t mean it’s untrue. • We need to be savvier about challenging scientists that take advantage of randomness to generate spurious results • "Depression is not currently routinely screened for after a patient has had a stroke," the author of the study said in an interview. "We think it ought to be, as treatment of depression could improve outcomes."

Causation, Correlation and Risk • Correlation is not always clear • Causation is often inferred • Risks are often over-sold What’s a Journalist to Do?

A journalist was responsible for an investigation of the scientific integrity of Wakefield’s work. • After his autism study was discredited, most media coverage about vaccinations reports “both sides of the story” about whether vaccines are safe or not. • However, the medical community almost universally endorses vaccine and believes that vaccines are safe. • Pockets of measles and croup due to vaccination refusal or lack of herd effect have been found in the U.S. and in the UK. Media Impact is Great • In 1998, Andrew Wakefield published a study on 12 children which was the basis for the belief that Autism is a result of vaccinations. • Press repeatedly reported these results, even though the scientific community was unable to reproduce the results. The existence of this study gave greater voice to other studies questioning safety of vaccination.

Basic Advice for a Journalist with Limited Time and Ideas Read as a skeptic at all times. Avoid most conclusions of causality. A lot can be understood by even a cursory read (<10 minutes) of the summary, the abstract, and the conclusion. Avoid the press release. The summary and abstract will tell you the results, but hardly ever hint as to what the limitations are. The conclusion will often tell you some caveats. Look up on PubMed.gov key words and see if other literature has been published on the topic – give other research equal time!

Research is routinely plagued Research is plagued What can a journalist do? Low levels of significance Multiple testing No acknowledgement of randomness in research design Lack of context/repeated experiments Scientists don’t know how to talk to journalists. But if you are looking to find one scientist willing to talk, you may not get the mainstream opinion. Write about the levels of significance, bias, caveats Ask the researchers about multiple testing. Did they adjust for them? Write about absolute risks. Look for a body of research rather than one specific paper Cite your sources! DON’T INDICATE CAUSATION WHEN A CORRELATION HAS BEEN SHOWN!

Lessons to be Learned Doting on Data There is no certainty, due to random effects. Don’t over-estimate the ability of poor data to give answers. Also, lots of data is unavailable. Risks need to be contextualized. Consensus is extremely important The world is complicated; many things interact with each other. The public voice is at least as loud as the scientific voice.

To Life!

Thank you!

Understanding Risk