Jamie Conrad Conrad Law & Policy Counsel SRA Annual Meeting Dec. 7, 2009

1. Research Funding & Scientific Integrity: Conflicts & CriteriaProposed Consensus Criteria for Assessing the Reliability of Scientific Work Jamie Conrad Conrad Law & Policy Counsel SRA Annual Meeting Dec. 7, 2009

2. Overview • Context/purpose for criteria regarding the reliability of scientific work • Proposed consensus criteria • Fora and mechanisms to advance the concept of reliability criteria

3. Context/Purpose • Policy makers need short-term proxies for the merits of scientific work – can’t wait for replication and general acceptance • Some proxies address validity/significance • I.e., ultimate correctness and importance • Think accuracy • E.g., weight of evidence methods or criteria

4. Context/Purpose • Others address reliability • I.e., faithfully reflect what was observed • Not product of error, unconscious bias or intentional manipulation • Think precision • Less ambitious than validity/significance; not sufficient, but necessary, esp. for policy purposes • I’m proposing a set of these criteria

5. Context/Purpose • Thesis: The more a given scientific work satisfies the following proposed reliability criteria, the more reliable it can be deemed. • Effectively a weighting scheme. • Not intended to set a minimum threshold for consideration, esp. in policy discussions • Unlikely that federal agencies in rulemaking can exclude a submission from any consideration

6. Proposed Criteria • Sources: Scientific community and federal agencies • Functions: • Undercut ability of sponsor or experimenter to skew the results or interpretation of research • Enable others to assess whether that might have occurred

7. Proposed Criteria(in rough priority order) • Whether the investigator has made full disclosure of sources of funding and other “competing interests” • Whether the investigator owns the research or testing results and is contractually guaranteed the right to release the data and to publish regarding it without prior sponsor approval • Whether the investigator has publicly released the research or testing data to allow others to review it and seek to replicate the results • Whether the investigator adhered to regulations or conventions establishing best practice standards in research • Whether the work was peer reviewed and the quality of that peer review • Whether, prior to its commencement, the study was included on apublic registry of research intended for use in policymaking • Where human subjects are involved, conformance to the Common Rule or comparable international protocols for the protection of human subjects • Whether the organization in which the investigator is located conducts or contracts for systematic oversight of investigators to ensure a culture of scientific integrity

8. Disclosure of ‘Competing Interests’ • Whether the investigator has made full disclosure of sources of funding and other “competing interests” • National Academies’ expectations are good example • Competing interests should not be the basis for discounting work, but rather an indication of honesty and a signal to those who care to examine the work more carefully

9. PI Ownership of Work • Whether the investigator owns the research or testing results and is contractually guaranteed the right to release the data and to publish regarding it without prior sponsor approval • Key indicator that research sponsor has taken the risk of funding truly independent work • Practices of American Chemistry Council’s Long-Range Research Initiative are good example

10. Public Release of Data • Whether the investigator has publicly released the research or testing data to allow others to review it and seek to replicate the results • Since replication is conventionally the ultimate test of scientific validity, action by an investigator to actively facilitate such replication supports the conclusion that the work is reliable • Obvious tension with PIs’ interests in future publications based on data, restraining free riding.

11. Adherence to Good Practices • Whether the investigator adhered to regulations or conventions establishing best practice standards in research • E.g., Good Laboratory/Epidemiology/Clinical Practices • These generally include written study protocols, often previously approved by government agencies; written standard operating procedures; transparent data management procedures; and publication of all results • Agencies like EPA and FDA generally require tests conducted in support of product approvals to comply with GLP regulations to ensure that the test results are fully auditable • Not typically followed in academic work • Principal limitation = GPs tend to lag behind method development

12. Peer Review • Whether the work was peer reviewed and the quality of that peer review • Journal peer review is usually sufficient to serve its academic purposes, but is not as searching or effective as regulatory peer review • OMB’s Peer Review Bulletin establishes minimum peer review standards for “influential scientific information” that federal agencies rely on or otherwise disseminate

13. Public Registries • Whether, prior to its commencement, the study was included on apublic registry of research intended for use in policymaking • Helps assure that sponsors do not (i) keep their efforts secret until they get the result they seek or (ii) change outcome measures after the fact • Registration of clinical trials now required by many medical journals, NIH, FDA, though questions being raised about extent of compliance • But could be adopted voluntarily in the public health area

14. Common Rule Compliance • Where human subjects are involved, conformance to the Common Rule or comparable international protocols for the protection of human subjects • In order to assure that the work will produce benefits to society that outweigh any risks to participants, these protocols necessarily include an assessment of the soundness of its design and methods

15. External Oversight of Research • Whether the organization in which the investigator is located conducts or contracts for systematic oversight of investigators to ensure a culture of scientific integrity • Greater confidence can be placed in an institution that arranges for its investigators’ specific research practices to be periodically and independent reviewed, either by a separate component of the institution or by an external entity

16. Advancing the Concept • Encouraging concordance among these criteria and • Miller/Hentges principles • BPC Recommendations 3 & 4 • Keystone Center Stakeholder Dialogue • GWU dialogue? • Possible convening bodies for formalizing consensus: • Private – SRA? • Government • OSTP? • Presidential Commission for the Study of Bioethical Issues • Hybrid – ACUS?

17. For More Detail • Carol J. Henry & James W. Conrad, Jr., Scientific & Legal Perspectives on Science Generated for Regulatory Activities, 116 EHP 136 (Jan. 2008)

18. James W. Conrad, Jr. Conrad Law & Policy Counsel 1615 L Street, NW, Suite 1350 Washington, D.C. 20036-5668 202-822-1970 202-822-1971 (fax) 703-405-1660 (cell) jamie@conradcounsel.com www.conradcounsel.com