Paradigms & Pragmatics: Researching Forensic Science Practice

1. Paradigms & Pragmatics: Researching Forensic Science Practice Robin Williams School of Applied Social Sciences University of Durham

2. Introduction • August 2001: ‘The research aims to improve knowledge of the effectiveness of the use of forensic science support within Durham Constabulary. In the longer term the study is concerned to understand the scope for increasing the contribution made by forensic science to the investigation of volume crime in the force area.’ • Since then: eight years of trying to see the animal through the undergrowth • The heterogeneity of ‘forensic science(s)’ • The multiple perspectives on ‘support’. • The complexity & relative invisibility of ‘investigations’ • The restless of the cognitive, legislative, judicial and policy contexts in which forensic science(s) practice is located • Some help from others but not enough

3. Necessity to research 2 kinds of social sites • Sites of Operation - where varying types of forensic science knowledge and practice are developed and implemented (crime scene examinations, ‘laboratories’, investigative briefings & debriefings, database entering and interrogating, etc.) • Sites of Deliberation - where legislation, regulation and policy on uses of forensic science is formulated, challenged and amended (parliamentary bodies, select committees, judicial authorities; expert and advisory bodies, regulatory agencies, government departments, etc)

4. CASE MATERIALS SUMMARY STATISTICS SCIENTIFIC & TECHNICAL INNOVATIONS ORGANISATIONAL AUDITS SITES OFOPERATION SITES OF DELIBERATION LEGISLATIVE INSTRUMENTS LEGAL JUDGEMENTS REGULATORY AND OTHER CODES OF PRACTICE MANUALS MEMORANDA OF UNDERSTANDING

5. CASE MATERIALS SUMMARY STATISTICS SCIENTIFIC & TECHNICAL INNOVATIONS ORGANISATIONAL AUDITS Networks of Actors & Agencies University Academics Expert Groups Commercial Actors Professional Policing Organisations Government Departments & Agencies Civil Society Groups SITES OFOPERATION SITES OF DELIBERATION LEGISLATIVE INSTRUMENTS LEGAL JUDGEMENTS REGULATORY AND OTHER CODES OF PRACTICE MANUALS MEMORANDA OF UNDERSTANDING

6. The US National Academies as a Site of Deliberation • The US National Academies: private, non-profit honorary societies of members elected by peers – the ‘honorary apex’ of US science. • National Academy of Sciences • National Academy of Engineering • Institute of Medicine • National Research Council – the ‘operating arm’ of the NAS • Provide authoritative advice on scientific matters to US Government and advises state and local governments • Reports on variety of topics commissioned by Government Agencies and Private Foundations • Successful ‘consensus reports’ frame policy and support new initiatives • Provides knowledge to justify regulatory decisions • Purifies ‘hybrid issues’ by separating politics from science • Previous Reports on Forensic Science • The Coroner and the Medical Examiner (1928) • DNA Technology in Forensic Science (1992) • The Evaluation of DNA Evidence (1996)

7. Strengthening Forensic Science InThe United States: A Path Forward • NRC Committee on Identifying the Needs of the Forensic Science Community. Drew on two standing committees: • Committee on Science, Technology, and Law • Committee on Applied and Theoretical Statistics • Committee formed in 2006, reported in 2009. • Chair and seventeen experts supported by fifteen NAS Staff ‘many talented professionals, some expert in various areas of forensic science, others in law, and still others in different fields of science and engineering.’ • Literature review, workshops and symposia and consultation with other specialists • Reviewed in draft form by 25 external experts including Barry Fisher ‘chosen for their diverse perspectives and technical expertise’.

8. Committee on Identifying the Needs of the Forensic Science Community • HARRY T. EDWARDS, (Co-chair), Judge, U.S. Court of Appeals for the District of Columbia • CONSTANTINE GATSONIS, (Co-chair), Director, Center for Statistical Sciences, Brown • MARGARET A. BERGER, Suzanne J. and Norman Miles Professor of Law, Brooklyn Law School • JOE S. CECIL, Project Director, Program on Scientific and Technical Evidence, Federal Judicial Center • M. BONNER DENTON, Professor of Chemistry, University of Arizona • MARCELLA FIERRO, Medical Examiner of Virginia (ret.) • KAREN KAFADAR, Rudy Professor of Statistics and Physics, Indiana University • PETE M. MARONE, Director, Virginia Department of Forensic Science • GEOFFREY S. MEARNS, Dean, Cleveland-Marshall College of Law, Cleveland State University • RANDALL S. MURCH, Associate Director, Research Program Development, Virginia Polytechnic Institute and State University • CHANNING ROBERTSON, Ruth G. and William K. Bowes Professor, Dean of Faculty and Academic Affairs, and Professor, Department of Chemical Engineering, Stanford University • MARVIN SCHECHTER, Attorney • ROBERT SHALER, Director, Forensic Science Program, Professor, Biochemistry and Molecular Biology Department, Eberly College of Science, The Pennsylvania State University

9. The Cultural Authority of the Report: Expertise & Credibility (Hilgartner) • Individuals as representatives of domains of expertise • Vouchsafed by associations with outstanding research institutions • Multidisciplinarity : comprehensive coverage & broad perspective • Uses two dominant registers: • An empiricist register – ‘allowing evidence to speak for itself’ ‘The findings of forensic science experts are vulnerable to cognitive and contextual bias.’ • Expert judgements – interpretative conclusions reached by qualified experts through rational deliberation ‘Between meetings, committee members reviewed numerous published materials, studies, and reports related to the forensic science disciplines, engaged in independent research on the subject, and worked on drafts of the final report.’ • Univocal narrative following extensive deliberation grounded in ‘the literature’ and ‘the evidence’ • Acknowledgement of non-scientific factors ‘It is also recognized that, given the complexity of the issues and the political realities that may pose obstacles to change, some recommendations will have to be implemented creatively and over time in order to be effective.’

10. The Cultural Authority of the Report: Controlling Information (Hilgartner) • Few details of commissioning background. ‘Recognizing that significant improvements are needed in forensic science, Congress directed the National Academy of Sciences to undertake the study that led to this report.’ • Committee and reviewer selection confidential ‘This Committee shall include members of the forensics community representing operational crime laboratories, medical examiners, and coroners; legal experts; and other scientists as determined appropriate.’ • Submissions, hearings and deliberations not directly reported. • A confidential review process • Visible agreement produced through invisible negotiation ‘The testimonial and documentary evidence considered by the committee was detailed, complex, and sometimes controversial. Given this reality, the committee could not possibly answer every question that it confronted, nor could it devise specific solutions for every problem that it identified. Rather, it reached a consensus on the most important issues now facing the forensic science community.’

11. Contesting the (Forensic) Credibility of NRC Reports • ‘Eminent scientists contributed their expertise to ensuring that DNA evidence offered in a courtroom would be valid and reliable (e.g., in the 1989 New York case, People v. Castro), and by 1996 the National Academy of Sciences had convened two committees that issued influential recommendations on handling DNA forensic science.’ • Airbrushes a major controversy surrounding 1992 Report • ‘emerged only after much strife, a threatened minority opinion and countless leaks of confidential drafts….described by NAS staff and committee members as one of the most contentious reports in recent years’ Science News Article • Significant and successful challenges by key institutional actors forced a second report on grounds that ‘the statistical and population genetic recommendations of the report have not been well received by the scientific community’ (Group letter from Weir and attendees at Second International Symposium on the Forensic Aspects of DNA, 1993) • Large differences between the disciplines represented on the two committees. • Documented in Aronson (2007)

12. Two Recurrent Issues: (i) Improving the ‘Scientific Foundations’ of Forensic Practice (a) • Endorses widespread observation of forensic science as practical research geared to tangible objectives set by other social institutions. In day-to-day forensic science work, the process of formulating and testing hypotheses is replaced with the careful preparation and analysis of samples and the interpretation of results. • Implicit characterisation of forensic science as largely ‘pre-normal’ science – absence of established, cumulative stable research tradition within particular domains ‘many forensic tests – such as those used to infer the source of toolmarks or bite marks have never been exposed to stringent scientific scrutiny’ • Emphasises heterogeneity: There is wide variability across forensic science disciplines with regard to techniques, methodologies, reliability, types and numbers of potential errors, research, general acceptability, and published material. • Technical innovations, especially in instrumentation, have far outdistanced any attempts to establish a theoretical framework for criminalistics as an autonomous discipline’ (Inman & Rudin Principles and Practice of Criminalistics page 64) • The well defined precision and accuracy of DNA has ‘set the bar higher for other forensic science methodologies’ (page 1-6) • Resonances of ‘paradigm shift

13. Two Recurrent Issues: (i) Improving the ‘Scientific Foundations’ of Forensic Practice (b) • A specific research proposal: to encourage the development of a body of knowledge derived from core scientific disciplines offering ‘data-based probabilistic assessments of the significance of claims’. • Valorises forensic genetics as exemplary: ‘The well defined precision and accuracy of DNA has set the bar higher for other forensic science methodologies’ • Avoids Saks and Koehler (2005) hyperbole: no endorsement of terms like ‘paradigm shift’ or ‘individualization fallacy’. • But note epistemic register: ‘Some forensic science methods have as their goal the “individualization” of specific types of evidence (typically shoe and tire impressions, dermal ridge prints, toolmarks and firearms, and handwriting). Analysts using such methods believe that unique markings are acquired by a source item in random fashion and that such uniqueness is faithfully transmitted’ • A method for reducing epistemic controversy but: what actors/agencies? What scientific disciplines will be relevant to which practice? What data generation and data sharing mechanisms will be supported and by whom?

14. Two Recurrent Issues: (ii) Standardizing Routine Forensic Practice (a) • Unremarkable endorsement of ‘standardization’ as a governing principle: ‘operational principles and procedures for many forensic science disciplines are not standardized or embraced, either between or within jurisdictions... accreditation of crime laboratories is not required in most jurisdictions. Often there are no standard protocols governing forensic practice in a given discipline. And, even when protocols are in place (e.g., SWG standards), they often are vague and not enforced in any meaningful way.’ • Standardization as socio-technical process necessary to the development and stability of scientific fields (Timmermans & Berg) • Design standards: explicit specifications of individual components of social and/or technical systems • Terminological standards: classification schemes that assure stability of meaning across different sites and times • Performance standards: specifications of what outcomes should be achieved • Procedural standards: guidelines or protocols specifying what should be done to perform a particular analysis properly

15. Two Recurrent Issues: (ii) Standardizing Routine Forensic Practice (b) • Standardization as a historically located social process: ‘Modernity can be viewed as a process of emphasizing technological standardisation and eliminating other established or culture based standards...Modern standard setting is characterised not by a change of type of standards but by the specificity of the processes created to prescribe them, and by the multiplicity of standards, their ubiquity and their formality’ (Krislov, 1997, quoted in Timmermans & Berg) • Another method for reducing epistemic controversy but: what technical and organisational features can be standardized? How are existing standards to be interpreted and new ones crafted? What institutions are involved or will be created to accomplish these ends? In what networks are such institutions located? What is the ‘reach’ of relevant standards?

16. Researching Forensic Science Practice (i) • What other (social) research issues/topics/questions concerning forensic science practice are engendered by the report? • Directly identified by its authors? • Research on ‘performance’ – on the reliability and accuracy of applications of forensic disciplines ‘reflecting actual practice on realistic case scenarios.’ • Research on observer bias and sources of human error, including studies of the effect on contextual bias • Research supporting the development of tools for advancing measurement, validation, reliability, information sharing, and proficiency testing in forensic science and to establish protocols for forensic examinations, methods, and practices. • Arise through consideration of its background, its focus and its recommendations? ‘

17. Researching Forensic Science Practice (ii) • Roy Porter (1995) Trust in Numbers: Objectivity in Science & Public Life. Two types of objectivity • Disciplinary objectivity • Typifies powerful specialist disciplinary communities • Values tacit knowledge • Artful application of insight based on learned experience • Disdain for standard solutions to complex problems • Mechanical objectivity • Trust in experts replaced by trust in mechanical rules, procedures and numbers: ‘Rules are a check on subjectivity; they should make it impossible for personal biases or preferences to affect the outcome of an investigation.. Mechanical objectivity stands for a rigorous method, enforced by disciplinary peers, canceling the biases of the knower….it lends authority to officials who have very little of their own’ • Mechanical objectivity • Both • Governance Issues About hybridity: what networks, what politics – uk/US. Salter paper? Public policy and network governance • For NRC: ‘The determination of uniqueness requires measurements of object attributes, data collected on the population frequency of variation in these attributes, testing of attribute independence, and calculations of the probability that different objects share a common set of observable attributes.’ • For Saks & Koehler: ‘The concept of individualization which lies at the core of numerous forensic science subfields exists only in a metaphysical or rhetorical sense. It has no scientific validity and is sustained largely by the faulty logic that equates infrequency with uniqueness’. • What are the trajectories of standardisation. Stnadrads provide order, but what is being ordred? Who is doing the ordering? What is the difffernce and how does it change routine practice?– and their limits • what agreements on standards are reached, by whom and how • What novel configurations of people and things are brought into being • What contingencies affect the performance of standard operating procedures • How do these play out in practice.How are routine and exceptional practices integrated into wider investitive practices ‘

18. Researching Forensic Science Practice (iii) • If this shift then why? • For Porter: epistemic shift engendered by outside pressure. Legitimacy questioned and authority shifts from character to method. • Hybridity of forensic science makes it chronically subject to such pressures. Dominated by issues of public credibility and socio-political legitimacy as much as by scientific foundations. Necessarily subject to legal stipulation and administrative corroboration. • Intensified by internal differentiation – DNA as Gold Standard/ Trojan Horse • If this shift, what kinds of additional researchable issues? • The emergence and positioning of new actors and agencies (e.g. UK Regulator, NPIA, SIPR? US NIFS) • The fate of prior verities (e.g. how is ‘individualization’ being rethought) (deliberative focus) • The estimation of professional value (e.g. efforts to understand and measure forensic science contributions to investigations and prosecutions) (operational focus)

19. The Fate of Prior Verities: Individualization • For NRC: ‘The determination of uniqueness requires measurements of object attributes, data collected on the population frequency of variation in these attributes, testing of attribute independence, and calculations of the probability that different objects share a common set of observable attributes.’ • For Saks & Koehler: ‘The concept of individualization which lies at the core of numerous forensic science subfields exists only in a metaphysical or rhetorical sense. It has no scientific validity and is sustained largely by the faulty logic that equates infrequency with uniqueness’.

20. The Estimation of Professional Value • What is the investigative ‘impact’ or ’significance’ of particular instances of forensic science analysis? • What Contribution: Identifying a suspect? Eliminating a suspect? Identifying a victim? Suggesting a line of inquiry? Curtailing a line of inquiry? Establishing a sequence of events? • What weight: Crucial? Significant? Corroborative? • Rigorous comparative case studies capable of considering the integration of forensic science analysis with other kinds of investigative knowledge and practice • Still conspicuous by their absence

21. Key references • Aronson, J. (2007) Genetic Witness: Science, Law and Controversy in the Making of DNA Profiling. New Jersey: Rutgers University Press. • Hilgartner, S. (2000) Science On Stage: Expert Advice as Public Drama. Stanford: Stanford University Press. • Lynch,M. et. al. (2008) Truth Machine: The Contentious History of DNA Fingerprinting. Chicago: Chicago University Press. • Porter, R. (1997) Trust in Numbers: Objectivity in Science and Public Life. Princeton: Princeton University Press. • Timmermans, S. (2006) Postmortem: How Medical Examiners Explain Suspicious Deaths. Chicago: University of Chicago Press. • Timmermans, S. & Berg, M. (2003) The Gold Standard: The Challenge of Evidence-Based Medicine and Standardization in Health Care. Philadelphia: Temple University Press.