evaluation of surveillance systems
Download
Skip this Video
Download Presentation
Evaluation of surveillance systems

Loading in 2 Seconds...

play fullscreen
1 / 56

Evaluation of surveillance systems - PowerPoint PPT Presentation


  • 130 Views
  • Uploaded on

Evaluation of surveillance systems. 17th EPIET Introductory Course Lazareto, Menorca, Spain September – October, 2011. Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada. Günter Pfaff. The surveillance loop. Health Care System. Public Health Authority. Reporting.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Evaluation of surveillance systems' - lita


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evaluation of surveillance systems

Evaluation of surveillance systems

17th EPIET Introductory Course

Lazareto, Menorca, Spain

September – October, 2011

Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada

Günter Pfaff

the surveillance loop
The surveillance loop

Health Care System

Public Health Authority

Reporting

Data

Event

Analysis & Interpretation

Decision

Information

Intervention

(Feedback)

importance of evaluation
Importance of evaluation
  • Obligation
    • Does the system deliver?
    • Credibility of public health service
  • In reality
    • Often neglected
    • Basis for improvements
  • Learning process
    • EPIET training objective
    • ”Do not create one until you have evaluated one”
slide4

Does the surveillance system…

  • Detect trends? Epidemics?
  • Provideestimatesofmorbidityandmortality?
  • Identifyriskfactors?
  • Stimulateepidemiologicresearch?
  • Assess effects of control measures
  • Leadto improvedclinicalpractice?
  • Lead to new/improved control measures?
  • Lead to betteradvocacyandincreasedfunding?
slide5

Criteria to look at

  • Simplicity
  • Flexibility
  • Acceptability
  • Data quality
  • Sensitivity and Predictivevaluepositive (PvP)
    • Capture-recapture
  • Representativeness
  • Timeliness

CDC guidelines

Footnote

slide6

Simplicity

Assimpleaspossiblewhilemeetingtheobjectives

  • Structure
    • Information needed
    • Number and type of sources
    • Training needs
    • Number of information users
  • Functionality
    • Data transmission
    • System maintenance
    • Data analysis
    • Information dissemination

Footnote

components of system
Components of system
  • Population under surveillance
  • Period of data collection
  • Type of information collected
  • Data source
  • Data transfer
  • Data management and storage
  • Data analysis: how often, by whom, how
  • Dissemination: how often, to whom, how

Confidentiality, security

slide9

Flexibility

  • Abilityofthesystemtoaccommodatechanges
    • New event to follow-up
    • New data about an event
    • New sources of information

Footnote

slide10

Acceptability

  • Willingnesstoparticipateinthesystem
    • Participation (%) of sources
    • Refusal (%)
    • Completeness of report forms
    • Timeliness of reporting
slide11

Acceptability

  • Factorsinfluencingthewillingnesstoparticipate
    • Public health importance
    • Recognition of individual contribution
    • Responsiveness to comments/suggestions
    • Time burden
    • Legal requirements
    • Legal restrictions

Footnote

data quality
Completeness

Proportion ofblank / unknown responses

Simple counting

Validity

True data?

Comparison

Records inspection

Patient interviews

...

Data quality
sensitivity
Sensitivity

= reported true cases total true cases

= proportion of true cases detected

slide15

Sensitivity

Disease

-

+

-

Total

notified

True +

False +

+

Notified

Total not notified

-

False -

True -

-

Total sick

Total not sick

Sensitivity = True + / Total sick

Specificity = True - / Total Not sick

PVP = True+ / Total notified

sensitivity versus specificity
Sensitivity versus specificity

The tiered system: confirmed, probable, possible

slide17

ConsequencesoflowPvP

  • Frequent "false-positive" reports
    • Inappropriate follow-up of non-cases
    • Incorrect identification of epidemics
  • Wastage of resources
  • Inappropriate public concern (credibility)

Footnote

measuring sensitivity
Measuring sensitivity
  • Find total true cases from other data sources
    • medical records
    • disease registers
    • special studies
  • Capture-recapture study
capture recapture
Capture-recapture
  • Used for counting total number of individuals in population using two or more incomplete lists
  • Originally used in wildlife counting(birds, polar bears, wild salmon…)
uses in epidemiology
Uses in epidemiology
  • Estimate prevalence or incidence from incomplete sources
  • Evaluate completeness of a surveillance system
principles
Principles
  • Two/more sources of cases with disease
    • Lists, registries, observations, samples
  • Estimate total number in the source population (captured and uncaptured) from the numbers of captured in each capture
assumptions
Assumptions
  • The population is closed
    • No change during the investigation
  • Individuals captured on both occasions can be matched
    • No loss of tags
  • For each sample, each individual has the same chance of being included
    • Same catchability
  • Capture in the second sample is independent of capture in the first
    • The two samples are independent, pYZ = pY pZ
slide23

Daddy, how many fish are in the aquarium?

Seaworld Oberhausen, August 2010

your options as a scientist
Your options as a scientist
  • Don‘t answer => Expect repeat question
  • Answer something => „How do you know?“
  • Consult an expert
  • Estimate yourself
meet the expert pulpo paul
Meet the expert - „Pulpo Paul“
  • Has nine brains and three hearts
  • Managed to predict all German games during the 2010 Football World Cup right
  • Predicted accurately the finale Netherlands-Spain

Binomial distributions only

slide26
http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Teshttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes
two source model
Two-source model

N=?

Y1

Source Y

Z1

Source Z

b

a

c

x=?

N= a + b + c + x

two source analysis
Two-source analysis

N = Y1 Z1 / a

Sensitivity of Y Ysn = Y/N = (a+c)/N

Sensitivity of Z Zsn = Z/N = (a+b)/N

how many persons are in the epiet 2011 introductory course
How many persons are in the EPIET 2011 Introductory Course?

Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“

how many persons are in the epiet 2011 introductory course36

Hand does not meet our case definition

This is our first view

1

4

4

5

2

3

4

4

3

How many persons are in the EPIET 2011 Introductory Course?

3

Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“, n=33

how many persons are in the epiet 2011 introductory course37

This is our second view

4

3

2

6

3

How many persons are in the EPIET 2011 Introductory Course?

Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=18

how many participants at the course
How many participants at the course?
  • Capture: Source ”View #1”
  • Recapture: Source ”View #2”
  • Estimations 
  • Assumptions hold? 
number of participants
Number of participants

Source View #2 – After Dinner Tutorial

Yes

No

Source View #1

Dinner

Yes

13

20

View #1

= 33

x

No

5

View # 2

= 18

N

= 13 + 20 + 5

+ x

N = 33 * 18 / 13 = 47

Sensitivity of View # 1 Sn1 = 33/47 = 70.2%

Sensitivity of View # 2 Sn2 = 18/47 = 38.3%

how many persons are in the epiet 2011 introductory course40

+ 2

This is our second view (revisited)

4

3

2

6

3

How many persons are in the EPIET 2011 Introductory Course?

Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=20

number of participants41
Number of participants

Source View #2, revised – After Dinner Tutorial

Yes

No

Source View #1

Dinner

Yes

13

20

View #1

= 33

x

No

7

View # 2

= 20

N

= 13 + 20 + 7

+ x

N = 33 * 20 / 13 = 51

Sensitivity of View # 1 Sn1 = 33/51 = 64.7%

Sensitivity of View # 2 Sn2 = 20/51 = 39.2%

so just how many are there
So, just how many are there?

9

9

9

25

18

30

2

5 off screen

Isla del Lazareto, Katharina‘s Lecture, Monday, 11 October 2010 – Case definition: “Persons in room“, N=53

assumptions may not hold
Assumptions may not hold
  • The population is closed
    • Usually possible
  • Individuals captured on both occasions can be matched
    • OK if good recording systems
  • For each sample, each individual has the same chance of being included
    • Rarely true
  • Capture in the second sample is independent of capture in the first
    • Rarely true
sources are independent most important condition
Sources are independent(most important condition)

Being in one source does not influence the probability of being in the other source

OR > 1 (positive dependence): underestimates N

OR < 1 (negative dependence): overestimates N

dependent sources
Dependent sources
  • Estimation of number of IVDU in Bangkok in 1991 (Maestro 1994)
  • Two sources used:
    • Methadone programme (April – May 1991)
    • Police arrests (June – September 1991)
  • Methadone  Need for drugs   Probability of being arrested  = negative dependence, overestimation of N
usefulness of capture recapture
Usefulness of capture-recapture
  • If conditions are met
    • Great potential to estimate population size by using incomplete sources
    • Cheaper than exhaustive registers or full counting
  • Two sources
    • Impossible to quantify extent of dependence
  • Multiple sources
    • Can adjust for dependence and variable catchability
examples of capture recapture
Examples of capture-recapture
  • STDs in The NL
    • Reintjes et al. Epidemiol Infect 1999
  • Foodborne outbreaks in France
    • Gallay et al. Am J Epidemiol 2000
  • Pertussis in England
    • Crowcroft et al. Arch Dis Child 2002
  • Invasive meningococcal disease
    • Schrauder et al. Epidemiol Infect 2006
slide49

Representativeness

  • A representative system accurately describes
    • Occurrence of a health event over time
    • Distribution in the population by place and time
  • Difficult to determine
    • Compare reported events with actual events
    • Characteristics of the population
    • Natural history of condition, medical practices
    • Multiple data sources
  • Related to data quality, bias of data collection, completeness of reporting

Footnote

timeliness
Timeliness

Analysis and interpretation

Reporting

Action taken

Disease onset

Diseasediagnosed

ofevent

Public Health Authorities

Clinician, labs

slide51

Buehler’sbalanceofattributes

Timeliness

Acceptability

Flexibility

Simplicity

Cost

Sensitivity

Representativeness

Predictive value positive

slide52

Improvingsurveillancesystems

  • Recommendations of evaluation
    • Continue
    • Revise
    • Stop
  • If revising
    • Increase participation rate of sources
    • Simplify notification
    • Increase the frequency of feedback
    • Broaden the net . . .
    • Activate data collection
slide53

Corollary

Carnunthum, Austria

Surveillance is like archeology of the immediate past –It requires your responsible imagination of an invisible reality.

literature
Literature
  • CDC. Updated guidelines for evaluating public health surveillance systems. MMWR 2001; 50 (RR-13): 1-35
  • WHO. Protocol for the evaluation of epidemiological surveillance systems. WHO/EMC/DIS/97.2.
  • Romaguera RA, German RR, Klaucke DN. Evaluating public health surveillance. In: Teutsch SM, Churchill RE, eds. Principles and practice of public health surveillance, 2nd ed. New York: Oxford University Press, 2000.
reading on capture recapture
Reading on capture-recapture
  • Wittes JT, Colton T and Sidel VW. Capture-recapture models for assessing the completeness of case ascertainment using multiple information sources. J Chronic Dis 1974;27:25-36.
  • Hook EB, Regal RR. Capture-recapture methods in epidemiology. Methods and limitations. Epidemiol Rev 1995; 17: 243-264
  • International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple-record systems estimation I: History and theoretical development. Am J Epidemiol 1995;142:1047-58
  • International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple-record systems estimation II: Applications in human diseases. Am J Epidemiol 1995;142:1059-68
ad