Evaluation of surveillance systems
Download
1 / 56

Evaluation of surveillance systems - PowerPoint PPT Presentation


  • 132 Views
  • Updated On :

Evaluation of surveillance systems. 17th EPIET Introductory Course Lazareto, Menorca, Spain September – October, 2011. Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada. Günter Pfaff. The surveillance loop. Health Care System. Public Health Authority. Reporting.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Evaluation of surveillance systems' - lita


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Evaluation of surveillance systems l.jpg

Evaluation of surveillance systems

17th EPIET Introductory Course

Lazareto, Menorca, Spain

September – October, 2011

Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada

Günter Pfaff


The surveillance loop l.jpg
The surveillance loop

Health Care System

Public Health Authority

Reporting

Data

Event

Analysis & Interpretation

Decision

Information

Intervention

(Feedback)


Importance of evaluation l.jpg
Importance of evaluation

  • Obligation

    • Does the system deliver?

    • Credibility of public health service

  • In reality

    • Often neglected

    • Basis for improvements

  • Learning process

    • EPIET training objective

    • ”Do not create one until you have evaluated one”


Slide4 l.jpg

Does the surveillance system…

  • Detect trends? Epidemics?

  • Provideestimatesofmorbidityandmortality?

  • Identifyriskfactors?

  • Stimulateepidemiologicresearch?

  • Assess effects of control measures

  • Leadto improvedclinicalpractice?

  • Lead to new/improved control measures?

  • Lead to betteradvocacyandincreasedfunding?


Slide5 l.jpg

Criteria to look at

  • Simplicity

  • Flexibility

  • Acceptability

  • Data quality

  • Sensitivity and Predictivevaluepositive (PvP)

    • Capture-recapture

  • Representativeness

  • Timeliness

CDC guidelines

Footnote


Slide6 l.jpg

Simplicity

Assimpleaspossiblewhilemeetingtheobjectives

  • Structure

    • Information needed

    • Number and type of sources

    • Training needs

    • Number of information users

  • Functionality

    • Data transmission

    • System maintenance

    • Data analysis

    • Information dissemination

Footnote


Components of system l.jpg
Components of system

  • Population under surveillance

  • Period of data collection

  • Type of information collected

  • Data source

  • Data transfer

  • Data management and storage

  • Data analysis: how often, by whom, how

  • Dissemination: how often, to whom, how

Confidentiality, security



Slide9 l.jpg

Flexibility

  • Abilityofthesystemtoaccommodatechanges

    • New event to follow-up

    • New data about an event

    • New sources of information

Footnote


Slide10 l.jpg

Acceptability

  • Willingnesstoparticipateinthesystem

    • Participation (%) of sources

    • Refusal (%)

    • Completeness of report forms

    • Timeliness of reporting


Slide11 l.jpg

Acceptability

  • Factorsinfluencingthewillingnesstoparticipate

    • Public health importance

    • Recognition of individual contribution

    • Responsiveness to comments/suggestions

    • Time burden

    • Legal requirements

    • Legal restrictions

Footnote


Data quality l.jpg

Completeness

Proportion ofblank / unknown responses

Simple counting

Validity

True data?

Comparison

Records inspection

Patient interviews

...

Data quality



Sensitivity l.jpg
Sensitivity

= reported true cases total true cases

= proportion of true cases detected


Slide15 l.jpg

Sensitivity

Disease

-

+

-

Total

notified

True +

False +

+

Notified

Total not notified

-

False -

True -

-

Total sick

Total not sick

Sensitivity = True + / Total sick

Specificity = True - / Total Not sick

PVP = True+ / Total notified


Sensitivity versus specificity l.jpg
Sensitivity versus specificity

The tiered system: confirmed, probable, possible


Slide17 l.jpg

ConsequencesoflowPvP

  • Frequent "false-positive" reports

    • Inappropriate follow-up of non-cases

    • Incorrect identification of epidemics

  • Wastage of resources

  • Inappropriate public concern (credibility)

Footnote


Measuring sensitivity l.jpg
Measuring sensitivity

  • Find total true cases from other data sources

    • medical records

    • disease registers

    • special studies

  • Capture-recapture study


Capture recapture l.jpg
Capture-recapture

  • Used for counting total number of individuals in population using two or more incomplete lists

  • Originally used in wildlife counting(birds, polar bears, wild salmon…)


Uses in epidemiology l.jpg
Uses in epidemiology

  • Estimate prevalence or incidence from incomplete sources

  • Evaluate completeness of a surveillance system


Principles l.jpg
Principles

  • Two/more sources of cases with disease

    • Lists, registries, observations, samples

  • Estimate total number in the source population (captured and uncaptured) from the numbers of captured in each capture


Assumptions l.jpg
Assumptions

  • The population is closed

    • No change during the investigation

  • Individuals captured on both occasions can be matched

    • No loss of tags

  • For each sample, each individual has the same chance of being included

    • Same catchability

  • Capture in the second sample is independent of capture in the first

    • The two samples are independent, pYZ = pY pZ


Slide23 l.jpg

Daddy, how many fish are in the aquarium?

Seaworld Oberhausen, August 2010


Your options as a scientist l.jpg
Your options as a scientist

  • Don‘t answer => Expect repeat question

  • Answer something => „How do you know?“

  • Consult an expert

  • Estimate yourself


Meet the expert pulpo paul l.jpg
Meet the expert - „Pulpo Paul“

  • Has nine brains and three hearts

  • Managed to predict all German games during the 2010 Football World Cup right

  • Predicted accurately the finale Netherlands-Spain

Binomial distributions only


Slide26 l.jpg

http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Teshttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes


Two source model l.jpg
Two-source modelhttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

N=?

Y1

Source Y

Z1

Source Z

b

a

c

x=?

N= a + b + c + x


Two source analysis l.jpg
Two-source analysishttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

N = Y1 Z1 / a

Sensitivity of Y Ysn = Y/N = (a+c)/N

Sensitivity of Z Zsn = Z/N = (a+b)/N


How many persons are in the epiet 2011 introductory course l.jpg
How many persons are in the EPIET 2011 Introductory Course?http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“


How many persons are in the epiet 2011 introductory course36 l.jpg

Hand does not meet our case definitionhttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

This is our first view

1

4

4

5

2

3

4

4

3

How many persons are in the EPIET 2011 Introductory Course?

3

Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“, n=33


How many persons are in the epiet 2011 introductory course37 l.jpg

This is our second viewhttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

4

3

2

6

3

How many persons are in the EPIET 2011 Introductory Course?

Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=18


How many participants at the course l.jpg
How many participants at the course?http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

  • Capture: Source ”View #1”

  • Recapture: Source ”View #2”

  • Estimations 

  • Assumptions hold? 


Number of participants l.jpg
Number of participantshttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

Source View #2 – After Dinner Tutorial

Yes

No

Source View #1

Dinner

Yes

13

20

View #1

= 33

x

No

5

View # 2

= 18

N

= 13 + 20 + 5

+ x

N = 33 * 18 / 13 = 47

Sensitivity of View # 1 Sn1 = 33/47 = 70.2%

Sensitivity of View # 2 Sn2 = 18/47 = 38.3%


How many persons are in the epiet 2011 introductory course40 l.jpg

+ 2http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

This is our second view (revisited)

4

3

2

6

3

How many persons are in the EPIET 2011 Introductory Course?

Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=20


Number of participants41 l.jpg
Number of participantshttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

Source View #2, revised – After Dinner Tutorial

Yes

No

Source View #1

Dinner

Yes

13

20

View #1

= 33

x

No

7

View # 2

= 20

N

= 13 + 20 + 7

+ x

N = 33 * 20 / 13 = 51

Sensitivity of View # 1 Sn1 = 33/51 = 64.7%

Sensitivity of View # 2 Sn2 = 20/51 = 39.2%


So just how many are there l.jpg
So, just how many are there?http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

9

9

9

25

18

30

2

5 off screen

Isla del Lazareto, Katharina‘s Lecture, Monday, 11 October 2010 – Case definition: “Persons in room“, N=53


The problem with the x finding a comprehensive view l.jpg
The problem with the X:http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/TesFinding a comprehensive view


Assumptions may not hold l.jpg
Assumptions may not holdhttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

  • The population is closed

    • Usually possible

  • Individuals captured on both occasions can be matched

    • OK if good recording systems

  • For each sample, each individual has the same chance of being included

    • Rarely true

  • Capture in the second sample is independent of capture in the first

    • Rarely true


Sources are independent most important condition l.jpg
Sources are independenthttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes(most important condition)

Being in one source does not influence the probability of being in the other source

OR > 1 (positive dependence): underestimates N

OR < 1 (negative dependence): overestimates N


Dependent sources l.jpg
Dependent sourceshttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

  • Estimation of number of IVDU in Bangkok in 1991 (Maestro 1994)

  • Two sources used:

    • Methadone programme (April – May 1991)

    • Police arrests (June – September 1991)

  • Methadone  Need for drugs   Probability of being arrested  = negative dependence, overestimation of N


Usefulness of capture recapture l.jpg
Usefulness of capture-recapturehttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

  • If conditions are met

    • Great potential to estimate population size by using incomplete sources

    • Cheaper than exhaustive registers or full counting

  • Two sources

    • Impossible to quantify extent of dependence

  • Multiple sources

    • Can adjust for dependence and variable catchability


Examples of capture recapture l.jpg
Examples of capture-recapturehttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

  • STDs in The NL

    • Reintjes et al. Epidemiol Infect 1999

  • Foodborne outbreaks in France

    • Gallay et al. Am J Epidemiol 2000

  • Pertussis in England

    • Crowcroft et al. Arch Dis Child 2002

  • Invasive meningococcal disease

    • Schrauder et al. Epidemiol Infect 2006


Slide49 l.jpg

Representativenesshttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

  • A representative system accurately describes

    • Occurrence of a health event over time

    • Distribution in the population by place and time

  • Difficult to determine

    • Compare reported events with actual events

    • Characteristics of the population

    • Natural history of condition, medical practices

    • Multiple data sources

  • Related to data quality, bias of data collection, completeness of reporting

Footnote


Timeliness l.jpg
Timelinesshttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

Analysis and interpretation

Reporting

Action taken

Disease onset

Diseasediagnosed

ofevent

Public Health Authorities

Clinician, labs


Slide51 l.jpg

Buehler’shttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tesbalanceofattributes

Timeliness

Acceptability

Flexibility

Simplicity

Cost

Sensitivity

Representativeness

Predictive value positive


Slide52 l.jpg

Improvinghttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tessurveillancesystems

  • Recommendations of evaluation

    • Continue

    • Revise

    • Stop

  • If revising

    • Increase participation rate of sources

    • Simplify notification

    • Increase the frequency of feedback

    • Broaden the net . . .

    • Activate data collection


Slide53 l.jpg

Corollaryhttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

Carnunthum, Austria

Surveillance is like archeology of the immediate past –It requires your responsible imagination of an invisible reality.


Thank you l.jpg
Thank you!http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes


Literature l.jpg
Literaturehttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

  • CDC. Updated guidelines for evaluating public health surveillance systems. MMWR 2001; 50 (RR-13): 1-35

  • WHO. Protocol for the evaluation of epidemiological surveillance systems. WHO/EMC/DIS/97.2.

  • Romaguera RA, German RR, Klaucke DN. Evaluating public health surveillance. In: Teutsch SM, Churchill RE, eds. Principles and practice of public health surveillance, 2nd ed. New York: Oxford University Press, 2000.


Reading on capture recapture l.jpg
Reading on capture-recapturehttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

  • Wittes JT, Colton T and Sidel VW. Capture-recapture models for assessing the completeness of case ascertainment using multiple information sources. J Chronic Dis 1974;27:25-36.

  • Hook EB, Regal RR. Capture-recapture methods in epidemiology. Methods and limitations. Epidemiol Rev 1995; 17: 243-264

  • International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple-record systems estimation I: History and theoretical development. Am J Epidemiol 1995;142:1047-58

  • International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple-record systems estimation II: Applications in human diseases. Am J Epidemiol 1995;142:1059-68