a blind analysis l.
Download
Skip this Video
Download Presentation
A Blind Analysis

Loading in 2 Seconds...

play fullscreen
1 / 48

A Blind Analysis - PowerPoint PPT Presentation


  • 88 Views
  • Uploaded on

A Blind Analysis. You are not allowed to peek!. Prof. Donald Koetke Senior Research Professor of Physics Valparaiso University. Outline. What is a “blind analysis”? What is a “bias”? Does “bias” = “systematic error”? What is the special (subtle) bias?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'A Blind Analysis' - jory


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
a blind analysis

A Blind Analysis

You are not allowed to peek!

Prof. Donald Koetke

Senior Research Professor of Physics

Valparaiso University

slide2

Outline

  • What is a “blind analysis”?
    • What is a “bias”?
    • Does “bias” = “systematic error”?
    • What is the special (subtle) bias?
  • What is the history of the blind analysis”?
  • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
  • Some thoughts and reflections --

VU Colloquium

slide3

Outline

  • What is a “blind analysis”?
    • What is a “bias”?
    • Does “bias” = “systematic error”?
    • What is the special (subtle) bias?
  • What is the history of the blind analysis”?
  • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
  • Some thoughts and reflections --

VU Colloquium

slide4

Outline

  • What is a “blind analysis”?
    • What is a “bias”?
    • Does “bias” = “systematic error”?
    • What is the special (subtle) bias?
  • What is the history of the blind analysis”?
  • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
  • Some thoughts and reflections --

VU Colloquium

slide5

Outline

  • What is a “blind analysis”?
    • What is a “bias”?
    • Does “bias” = “systematic error”?
    • What is the special (subtle) bias?
  • What is the history of the blind analysis”?
  • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
  • Some thoughts and reflections --

VU Colloquium

slide6

Outline

  • What is a “blind analysis”?
    • What is a “bias”?
    • Does “bias” = “systematic error”?
    • What is the special (subtle) bias?
  • What is the history of the blind analysis”?
  • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
  • Some thoughts and reflections --

VU Colloquium

slide7

Outline

  • What is a “blind analysis”?
    • What is a “bias”?
    • Does “bias” = “systematic error”?
    • What is the special (subtle) bias?
  • What is the history of the blind analysis”?
  • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
  • Some thoughts and reflections --

VU Colloquium

slide8

A Blind Analysis

What is a “bias”?

Does “bias” = “systematic error”?

What is the special (subtle) bias?

“A prejudice in favor of or against” -- may bedue to computer codes, equipment performanceor setting, decisions/selections/cuts imposed, etc.

Yes - if it causes a systematic shift in a result(This is not a “mistake” or “blunder”.)

Experimenter biasin making decisions/choices to achieve a desired answer. Examples …

VU Colloquium

slide9

A Blind Analysis

You do an experiment and…

You know what the answer “should be” -i.e., you know what the “accepted” answer is -- but your answer is different -outside of errors.What do you do?

Asymmetry!!

You make a measurement and within errorsit agrees with the “accepted” answer --What do you do?

VU Colloquium

slide10

A Blind Analysis

The goal of a “blind analysis” is to preventthe experimenter from (unconsciously - or consciously) making decisions in the analysis that would affect the result based on:

  • The predictions of a model or theory (e.g., the Standard Model for particle physics)
  • Previous measurements known to the experimenter
  • The experimenter’s intuition or other predisposition

These are all examples of personal bias

VU Colloquium

slide11

Computer codes have been developed and tested

  • Decisions about the number of events (trials) needed have been made (cuts have been selected)
  • Apriori agreements are reached about what to do when the real answers from the experiment are revealed; no further analysis!

A Blind Analysis

What is a “blind analysis”?

A “blind analysis” is an analysis of measured data in which the final answer is kept hidden from the experimenters until all of the decisions about the analysis have been made:

VU Colloquium

slide12

Outline

  • What is a “blind analysis”?
    • What is a “bias”?
    • Does “bias” = “systematic error”?
    • What is the special (subtle) bias?
  • What is the history of the blind analysis”?
  • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
  • Some thoughts and reflections --

VU Colloquium

slide13

Medical research -- blind tests!

Patients don’t know whether they are getting --

the medicine/treatment, or,

a placebo

Patients are assigned to (a) or (b) ramdomly

History

Blind analysis begins in ~1930s with -

Therefore - the patients are “blind” to their treatment

They may imagine symptoms or cure, etc., but only the researcher/physician knows if these can be due to the medicine/treatment.

VU Colloquium

slide14

History

Blind analysis begins in ~1930s with -

Medical research -- blind tests!

  • Patients talk to medical researcher

e.g., how are you feeling? is pain less or more? are the other problems? serious or not so serious? before or after you take the medicine? …etc.

  • Researcher hears what patient says
  • Researcher examines the patient
  • Researcher records all this as “data”

Do you see a problem here?

c.f.,1937 JAMA 26 June 2178/2.

VU Colloquium

slide15

History

Blind analysis begins in ~1930s with -

Medical research -- blind tests!

The patients are “blind” to their treatment - but,

the researcher is NOT!

  • Therefore, the medical researcher can (and will) unconsciously and unintentionally interpret and record the information (data) with this bias (knowledge).

The solution is…

VU Colloquium

slide16

1948 Am. Heart Jrnl., XXXVI, 529.

1950 Am. Jrnl. Med., IX, 142/1.

History

Blind analysis begins in ~1930s with -

Medical research -- The double blind test!

The patients are “blind” to their treatment - and,

the researcher is “blind” to who is getting treated!

  • The list of patients and their treatment is prepared and maintained by someone who is NOT participating in the research. The list is sealed in the “black box”.
  • The list can be retrieved from the “black box” only after all the analysis is completed.

VU Colloquium

slide17

History

Blind analysis begins in ~1930s with -

Medical research -- The double blind test!

Consider a simple analysis -- “Was the medicine effective?”

  • The data on all patients will indicate whether the medication was effective for each patient.
  • When this data analysis is complete (and free from mistakes) and any disagreement among the researchers have been settled and any concerns about bias have been removed, i.e., then (and only then) -
  • The “black box” can be opened and the list can be retrieved.
  • The data on all patients is now grouped into two groups: group (a) and group (b)

VU Colloquium

slide18

History

Blind analysis begins in ~1930s with -

Medical research -- The double blind test!

The simple analysis -- “Was the medicine effective?”

  • For what fraction of group (a) [fa] and group (b) [fb] was the medicine effective?
  • Do a statistical analysis to determine whether the difference in these two fractions is statistically significant - or whether it is consistent with a random occurrence.The result is now known!
  • The researchers agree to accept the results obtained and no further analysis of the data is permitted - unless there has been a blunder (mistake) - in which case fix the blunder and report both results and the nature of the “fix”.

VU Colloquium

slide19

History

Blind analysis has become the standard methodology in clinical trials.

Blind analysis has been used in the physical sciences only in recent years.

Physicists(and astronomers) are, of course ---- Careful-- Quantitative-- Attentive to bias (systemtic errors)

Why should they need a blind analysis?

VU Colloquium

slide20

History

Ernest Rutherford (1934) --

“It seems to me that in some way it is regrettable that we had a theory of the positive electron before the beginning of the experiments. Blackett* did everything possible not to be influenced by the theory, but the way of anticipating results must inevitably be influenced to some extent by the theory. I would have liked it better if the theory had arrived after the experimental facts had been established.”Ernest Rutherford, Proc. Solvay Conference, (Gauthier-Villars, Paris 1934), p 177.*Nobel prize in physics for discovering the positron (1948)

VU Colloquium

slide21

In every case the data agreed with the theoretical

ratios within less than the standard errors.

Taking the whole together, 2 was 41.6 on

84 degrees of freedom, giving a probability that hewould have measured this well to be only7x10-5 !!

History

Gregor Mendel (1865) --

The classic case is Gregor Mendel’s

work on inheritance.

VU Colloquium

slide22

History

Modern examples --

From nuclear and subnuclear physics ---

Reason: It’s the field in which the blind analysis techniques have been widely used - and -

It is the field with which I am most familiar.

VU Colloquium

slide23

History

VU Colloquium

slide24

History

VU Colloquium

slide25

B-meson lifetime ratio

The mean ratio has a 2 of 4.5 for 13 degrees of freedom; P=0.985

History

8 Major experiments

Over 2000 physicists

Bottom line:

The agreement appears to be too good!!

VU Colloquium

slide26

Outline

  • What is a “blind analysis”?
    • What is a “bias”?
    • Does “bias” = “systematic error”?
    • What is the special (subtle) bias?
  • What is the history of the blind analysis”?
  • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
  • Some thoughts and reflections --

VU Colloquium

slide27

An example of a “search” for an uncommon occurrence

A reaction that violates the Standard Modelconservation of lepton number

If you do not find the reaction, you have not shown the Standard Model to be incorrect

If you do find the reaction, you had better be very sure that you have got it right!!!

VU Colloquium

slide28

Signal: Ee = E = 52.8 MeV

Boxes represent 2 boundaries

Examples

MEGA

Search for   e

Question: When and How are the values of  determined?

VU Colloquium

slide29

Signal: Ee = E = 52.8 MeV

Boxes represent 2 boundaries

Examples

MEGA

Search for   e

Blind Analysis: The boundaries aredetermined before the analysis of (most of) the data!

VU Colloquium

slide31

Examples

p = 29.8 MeV/c

Measure: pe()

TWIST

High precision   ee

A search for physics beyond (not included in) the Standard Model.

VU Colloquium

slide32

Analyzed expimentaldata

Nexp(x) = a’ + b’x

Simulation ofthe experimentusing a,b

Number

N(x) = a + bx

a,b from theory

X = E/Emax

Examples

Finda’ &b’ bycomparison ofreal datawithsimulateddata

The detector system distorts the distribution

Simulation = “Monte Carlo”

VU Colloquium

muon decay spectrum

Examples

Muon decay spectrum

Current SM

 = -0.007 ± 0.013 0

 = 0.7518 ± 0.0026 3/4

 = 0.7486 ± 0.0026 ± 0.0028 3/4

P = 1.0027 ± 0.0079 ± 0.0030 1

VU Colloquium

slide35

TWISTmeasured spectrum

Standard Model spectrum

Monte Carlo computer code

,,,

Energy

Energy

Cos()

Cos()

Examples

VU Colloquium

slide36

Compare spectra  ’,’,’,’

TWISTmeasured spectrum

TWISTsimulated spectrum

’,’,’,’

,,,

Energy

Energy

Cos()

Cos()

Examples

Problem!

This spectrum is the S.M. spectrum -- nothing is hidden; the experimenter is not “blinded”.

VU Colloquium

slide37

TWISTmeasured spectrum

Standard Model spectrum

Monte Carlo computer code

o,o,o,o

Energy

Energy

Cos()

Cos()

Examples

Unknown spectrum

o,o,o,o are generated randomly, are encrypted, stored secretly, and used to generate the simulated data. Nobody knows what the offsets from ,,, are.

VU Colloquium

slide38

TWISTmeasured spectrum

TWISTsimulated spectrum

?

Energy

Energy

Cos()

Cos()

Examples

Compare spectra to get  , , , 

’,’,’,’

 = ’ - o = ’ - o = ’ - o = ’ - o

VU Colloquium

slide39

o,o,o,o

Examples

  • Before you can open the black box:
  • Get all computer codes working and tested
  • Identify all sources of systematic error and evaluate the size of each one
  • Take all of the data you will need including data to help estimate the systematic errors
  • Be sure that the Monte Carlo programs accurately simulate your experiment or you will have false values
  • Analyze all of the data you intend to use to get the result

VU Colloquium

slide40

o,o,o,o

Measure

Calculate the results

Examples

 = ’ - o = ’ - o = ’ - o = ’ - o

Compare with Standard Model predictions

Write the paper

VU Colloquium

slide41

Examples

Let’s look at one of your experiments

from PHYS-245

VU Colloquium

slide42

2

1

osc

D

Examples

A measurement of the speed of light

Present accepted value: c = 299,792,458 m/sc  3 x 108 m/s

VU Colloquium

slide43

2

1

osc

D

Examples

Blinded

A measurement of the speed of light

Present accepted value: c = 299,792,458 m/sc  3 x 108 m/s

VU Colloquium

slide44

Outline

  • What is a “blind analysis”?
    • What is a “bias”?
    • Does “bias” = “systematic error”?
    • What is the special (subtle) bias?
  • What is the history of the blind analysis”?
  • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
  • Some thoughts and reflections --

VU Colloquium

slide45

Reflections

Guide to a blind analysis:If my answer were to come out to be six standard deviations from the expected result, what would I do? Make the list, and then -do all of that before you look at the answer!

A blind analysis is intended to guard against experimenter bias. It will not guard against fraud; that requires integrity and honesty.

A blind analysis is likely not the fastest way to an answer.

VU Colloquium

slide46

Reflections

It is not always possible to achieve perfect “blindness” -- e.g., drug testing.

Don’t need to plan everything in the analysisbefore beginning; just keep the answer hidden.

A blind analysis may not work for every experiment -but it is worth investigating before you begin.

In a blind analysis you want to hide the answer from anyone else who might want to offer advice that may be based on the answer.

VU Colloquium

slide47

Reflections

Thankyou

A blind analysis removes the “comfort factor” - knowing what answer you are getting so you can make changes, do more analysis, repeat measurements, etc., if the answer is not what you expect.

But, that may not be the best science!

VU Colloquium