Tools for Evidence: Building a Quality Dataset

1. Tools for Evidence: Building a Quality Dataset Rita E. Adkins, M.P.A. Matthew G. Hile, Ph.D. Keith Eldridge, B.S.Missouri Institute of Mental Healthmadkinr@mimh.edu COSP is a cooperative agreement funded by the Center for Mental Health Services of the Substance Abuse and Mental Health Services administration, U.S. Department of Health and Human Services

2. Tools for Evidence • What is quality evidence? • Tools for building quality evidence • Threats to quality evidence • COSP Data Collection • COSP Data Entry • COSP Data Cleaning • COSP Data Distribution • COSP Data Monitoring

3. What is Quality Evidence? • The foundation of good evidence in a research project is rigorous planning • Planning in the early stages is well worth the investment of time and resources • Data quality is evidenced by valid and reliable data

4. Threats to Quality Data • Literature has suggested that up to 1% of data is in error, which could impact findings • Not all error is random • Systematic error can often be attributed to: • lack of resources • lack of skills or knowledge • lack of attention to details • lack of clearly defined data elements

5. Tools for Building Quality Evidence • Development of study protocol to capture measurable, meaningful data • Standardization of the interviewing process: • Continuous Interviewer training: Arndt, et. al. (1994) found that 2.4% of their data were in error, the majority from the interviewers • Standardization of data capture: • Automated entry systems with built-in error-trapping • Systematic scoring of scales • Careful analysis of data for logical inconsistencies • Development of monitoring systems

6. Building the COSP Data Repository-Data Collection Having a protocol with psychometrically established scales is critical to efficacy of data • COSP Protocol was collaboratively developed by the Steering Committee • Instruments were chosen based on their reliability and validity, with established psychometric properties • Pilot study was conducted and analyzed before final protocol was released

7. Building the COSP Data Repository-Data Collection Standardized interviewing methods insure that findings are related to interventions rather than collection variations • Standardized Interviewer Procedures • Development of a Training Manual and Video • Pre-post test for Manual • Development of a Q by Q Manual • Extensive training workshops • Ongoing training at the site level • Continuous Quality Improvement: • Listserv for interviewer questions & concerns • Review of taped interviews

8. Building the COSP Data Repository-Data Entry System “Garbage in, Garbage out” Designing a data entry system can reduce overall error rate • Designed to catch errors • Requires double entry • Built-in skip patterns and consistency checks • Does not allow • Duplicate records • Out-of-Range responses • Null fields

9. Building the COSP Data Repository-Data Entry System • Developed in Access 97 • Designed for “heads down” data entry • Entry at the site level • Security system • Contains over 1,700 variables • Multiple data points in one database

10. Building the COSP Data Repository-The Multisite Repository The complete dataset in the data repository is a resource for present and future analysis • Data electronically transferred via the internet from 7 sites across the country • Data merged into 1 dataset • Over 1700 data elements • Over 13,000,000 observations • Data consistency capabilities through queries

11. CP CP CP CP Site 1 Site 2 Site 3 Site n • Standard Database • Includes Constraints • Standard Entry Program • Logic validations and additional constraints • Entry verification • Error Types • Recording • Entry • Data Formatting • Illogical data • …………. Data Data Data Data entry entry entry entry Errors Repository Analysis COSP Data Quality Protocol

12. Building the COSP Data Repository-Data Cleaning System • Logic flow analysis developed to determine: • Dates are consistent and possible (i.e., Date of Hospitalization > DOB) • Group assignment consistent across waves • Racial group endorsed is consistent with “preferred ethnic identity” • On-going process to ID potential problem areas Despite proactive efforts, errors will occur

13. Building the COSP Data Repository-Preparing the Data for Public Use Archiving of data reinforces open scientific inquiry, encourages diversity of opinions, improves methods of data collection, further knowledge development, and allows for the testing of new methods (Feinberg, 1994) • Currently still collecting data • Naming conventions and codebooks automatically generated by Access database • The COSP project data collection design incorporates the characteristics for dataset integrity defined by the Inter-University Consortium for Political and Social Research (ICPSR) • Identifiers will be removed before archiving

14. MIMH Files from 7 Sites Data Entry Merged Files Multiple Files Transmitted to CC Data Analyzed Merged Files ROW Data Trail Common Protocol

15. Building the COSP Data Repository-Study Progress Monitoring • On-going monitoring of standardized interviewing • Interviewer Alerts • Updates to the Question by Question Manual • Follow-up windows monitored • Patterns of missing variables tracked • Collection of diagnoses monitored

16. Building the COSP Data Repository-Study Progress Monitoring • Site Reports to Coordinating Center • Weekly Flash Report • Track enrollment by condition • Quarterly Report • Report on recruitment, enrollment, and attrition • Report cost data status • Narrative on site accomplishments, problems encountered, and goals

17. Building the COSP Data Repository-Study Progress Monitoring • Coordinating Center Reports • Enrollment • Engagement • Cost data • Projections • Power analysis • Baseline equivalencies • Other summaries

18. This has been anMIMH production