Getting better all the time?

1. Dave Walsh & Mick King, University of Derby, UK. Getting better all the time?

2. Aim of study: To examine how well investigators evaluate their own interview skills Post-PEACE: (some) areas of interviewing have improved (Clarke & Milne, 2001; Griffiths & Milne, 2006; Griffiths, 2008; Leahy-Harland, 2011; Soukara, Bull, Turner, Vrij, & Cherryman, 2009; Shawyer, 2009; Walsh & Bull, 2010; 2012; Walsh & Milne, 2008) iNTRODUCTION

3. Problems with…. • Training alone will not lead to sustained improved performance(Holton, Bates, Seyler, & Carvalho, 1997; Rouiller & Goldstein, 1993; Yammill & Mclean, 2001). • with supervision (Clarke & Milne, 2001; Stockdale, 1993; Walsh & Milne, 2007) • expectations after training (Shepherd & Griffiths, 2013) • understanding what is good interviewing (Cherryman, 2000; Walsh & Bull, 2011) Reasons for continuing indifference in performance

4. Self evaluation involves “a high level of self-awareness and the ability to monitor one’s own learning and performance” (Cassidy, 2006, p. 170). Highly demanding skill (Argyris & Schön; 1978) Persistence in undertaking the task enables effective self-evaluation (Hanrahan & Isaacs, 2001; Stefani, 1998) and perceptions of improved performance (Walsh & Milne, 2007) Importance of self –evaluation

5. Investigators from GLA – entire investigation personnel Undertook a training exercise involving a simulated interview Measured for skill levels using an assessment scale (46 dimensions) – evolved from Bull & Cherryman, 1995; Clarke & Milne, 2001; Griffiths, 2008; Walsh & Bull, 2010; Walsh & Milne, 2008) Dichotomous - and Likert scales; 1 – poor , 5 – highly skilled Overall interviewing skill Methodology

6. By participants themselves at end of training from video-taped recording • By researchers after training from video-taped recording • Between researchers to ensure inter-rater reliability Stages of measurement

7. Between experts Assessor 2 examined 50% of Assessor 1’s sample (N=9) All correlations significant (where p = ≤.01) at ≥ .82 except ‘Overall interview’ score; 0.64, p = <0.05. Inter-rater scores

8. Between expert and investigator (self –assessed) • Correlations significant • Appropriate structure 0.71, p= <0.01 • Topic development (0.49), Points to prove (0.54), Conversation management (0.59), Cognitive interview (0.49), Intermittent summaries (0.46)- all significant; p = <0.05 • All other correlations (inc. ‘overall assessment) scored no more than 0.22, all being n.s., except ‘Self confidence’; 0.45,n.s. Inter-rater scores

9. Mean score (N=18) • Self assessment Expert assessment • Planning/preparation 3.63 (0.65) 3.39 (0.61) • Caution delivery * 4.59 (0.58) 4.06 (0.87) • Checks understanding of caution ** 4.27 (0.76) 2.72(1.18) • Explains interview purpose** 3.68 (1.16) 2.50 (0.99) • Opportunity to provide account explanation** 3.27 (1.47) 1.17 (0.39) • Rapport* 3.82 (1.84) 3.11(1.24) • Deals with lack of co-operation** 3.86 (0.70) 2.61 (0.78) • Appropriate interview structure** 4.05 (0.83) 3.17 (0.99) • Encourages comprehensive account** 3.96 (0.57) 3.06 (0.94) • Provides appropriate questioning strategy** 3.73 (0.76) 2.78 (0.88) • Provides periodic summaries and linkages ** 3.36 (1.08) 1.56 (0.62) • Examines points to prove and defences 3.23 (1.05) 2.83 (0.79) • Explores information received* 3.69 (0.77) 3.00 (1.14) • Identifies and tests inconsistences * 3.27 (1.15) 2.56(0.52) • Challenges appropriately** 2.77 (1.33) 2.17(0.38) • Explores intent and motive** 2.59 (1.21) 1.33 (0.59) • Uses pauses** 2.95 (1.24) 1.28 (0.67) • Conversation management* 3.67(1.08) 3.06(0.54) • Cognitive interview skill** 2.83 (1.84) 1.17 (0.38) • Self confidence 4.00 (0.81) 3.78 (0.55) • Open mindedness 3.87 (0.88) 3.56 (0.78) • Flexibility** 3.91 (0.91) 2.94 (0.64) • Communication skills ** 4.05 (0.78) 3.33 (0.49) • Active listening** 3.83 (0.62) 3.11 (0.76) • Advises what happens next * 2.27 (1.67) 1.39 (o.70) • Provides final summary* 2.55 (1.52) 1.61 (0.98) • Overall assessment * 3.46 (0.67) 2.71(0.68)*** • Ratings significant **p ≤0.01; *p ≤0.05 • *** both assessors scores skill levels –self assessed v. expert

10. Wilcoxon Signed Rank test resultsz r • Planning/preparation 1.73 0.29 • Caution delivery* 2.00 0.33 • Checks understanding of caution** 3.09 0.52 • Explains interview purpose** 2.55 0.43 • Opportunity to provide account explanation** 3.23 0.54 • Rapport* 2.00 0.33 • Deals with lack of co-operation** 3.18 0.53 • Appropriate interview structure** 3.31 0.55 • Encourages comprehensive account** 3.00 0.50 • Provides appropriate questioning strategy** 2.00 0.33 • Provides periodic summaries and linkages **3.57 0.60 • Examines points to prove and defences 1.54 0.26 • Explores information received * 2.12 0.35 • Identifies and tests inconsistences* 2.32 0.39 • Challenges appropriately** 2.63 0.44 • Explores intent and motive** 2.85 0.48 • Uses pauses** 3.04 0.51 • Conversation management* 2.31 0.39 • Cognitive interview skill** 3.58 0.60 • Self confidence 1.15 0.19 • Open mindedness 1.13 0.19 • Flexibility** 2.93 0.48 • Communication skills** 2.48 0.41 • Active listening** 2.67 0.45 • Advises what happens next* 1.61 0.27 • Provides final summary* 1.90 0.32 • Overall assessment * 1.50 0.25 • Ratings significant **p ≤0.01; *p ≤0.05 skill levels –self assessed v. expert

11. Investigators seemingly tend to self-enhance Exaggerated scores after training Some tend to be overly self-critical Few accurate self-assessments Self-evaluation as a basis for self-improvement? Implications for prActice

12. Investigators submitted 2000 word reflective self- assessments To be compared to their/our rating scales Interview skills to be measured again 12 months after training from field interviews Also questionnaire relating to self-evaluation task Is there improvement/deterioration/maintenance in performance levels Is there any association between performance (improved/deteriorated/maintained) levels and self-reports of undertaking self-evaluation task future developments