Evaluating Information & Knowledge Services Using Narrative Techniques - a case study

1. Evaluating Information & Knowledge Services Using Narrative Techniques - a case study Nerida Hart Chair, Knowledge Management Division Special Libraries Association Information Online 2009

2. Presentation Outline • What was this project all about? • Potential solutions • What are narrative techniques? • Knowledge & Information Services approach • How K&IS used narrative techniques • Limitation • Results • Conclusions

3. Background 2005 • managing a special library (knowledge service) with approximately 40,000 clients • 5 agencies being serviced - FaCS, Centrelink, Child Support Agency, Social Security Appeals Tribunal, Indigenous Affairs • > 40 staff grouped into 5 service delivery sections

4. What was the problem we needed to solve? • We were due for the dreaded review !!!! • How did we prove our worth within our client groups? • Time to do this was limited and we didn’t want to boil the ocean but concentrate on our biggest value add service - “research services”

5. How did we go about proving our worth… • Library statistics only measure input and output and we needed to show how we contributed to client outcomes • Need to cater for emergent benefits (the real benefits of the research service) • What were the real experiences of our clients • Need to encourage good practices and disrupt poor practices • Looking for ways of continuously improving products and services

6. Potential methodologies • Surveys - great for the number devoted managers who need ‘hard’ evidence • Focus groups - can be manipulated • Interviews - time was against us • Narrative - excellent - would give us ‘depth’ but which narrative technique to use

7. Cynefin Complexity Framework

8. Why Anecdote Circles as a narrative technique? • Anecdote circles use anecdotes of personal experiences of workers to gain evidence of what is really happening in a complex organisation • They are more open-ended than focus groups and allow for the unexpected. • Focus groups are more concerned with opinions and judgements than anecdotes which are more concerned with personal experiences • Anecdotes reveal the values and behaviours of the staff – make sense of the organisation’s culture • Anecdotes link events in a meaningful way

9. Components of Narrative Approach using Anecdote Circles • Employing Narrative approach required dividing the work into 4 phases: • Preparation (1/2 day) • Discovery – Anecdote circles (90-120 minutes each) • Sensemaking (1 day) • Intervention Design – using complexity principles (i.e. action planning and implementation) (1/2 to 1 day)

10. Knowledge & Information Services Approach Project was part of project management certification for a staff member • Parallel survey – as a control group • Narrative – use of anecdotes • Full skills and knowledge transfer from our consultants Anecdote Pty Ltd (www.anecdote.com.au)

11. Use of Narrative in K & IS • Preparation: • Identification of themes: • Client information seeking behaviour • Quality • Accuracy and authenticity • Client experiences with research staff

12. Discovery • Discovery – Anecdote circles • Four sessions including one regional session • 28 participants in total (6-12 per session) • All sessions recorded and transcribed • Anecdotes extracted from the transcripts • All participants were not personally identified in the report

13. “We were thinking about FaCS policy priorities. The bibliography the library sent me, at very short notice, was concentrated on sources like the OECD and other government publications, because I think there was an acknowledgement that it we were going to try to introduce it to our Department we had to have sources that would be credible.” Survey results vs Anecdotes

14. Anecdote example : Depth of Information “When I got the material it was not really what (I wanted). In the end I didn’t use very much of what was suggested to me. Given that I didn’t go back and define more closely and try to work it through with the person… so it was a time factor more than anything else. I would have gone back. In the end it was easier for me to go the way I did go.”

15. Sensemaking • Sensemaking • One day workshop • Used an independent facilitator • Objectivity • Distancing ourselves from the results and emotional involvement • Enables future credibility • Identifying themes, behaviours and archetypes

16. Interventions / Action Planning • Interventions • Marketing for greater awareness of our services • Follow up after research completed • Clarification of client research requirements • Revamping parts of our intranet site • Enhancing our role as an information and knowledge sharing catalyst (connectors)

17. Limitations • Geographic dispersal of the clients • Getting the numbers and timing right – availability of participants (Welfare to Work priorities of FaCS and Centrelink) • Christmas looming • Convincing our clients that this was important – benefits to both us and them

18. Conclusions • Benefits to K&IS and our clients: • Tangible evidence of K&IS contribution to core business through real examples of time saving, authenticity of information, contribution to the big picture goals of the client agencies • Skills and knowledge development of K&IS staff • Stronger relationships with K&IS clients and increase in return business • Improvements in our training methods for K&IS clients • Improvements in research processes which then benefits the client groups

19. References • How to run this process www.rkrk.net.au • Anecdote Pty Ltd www.anecdote.com.au • Dave Snowden’s Cynefin Framework www.cognitive-edge.com

20. Questions and Comments Contact details: nerida.hart@lwa.gov.au Linked in: www.linkedin.com/in/neridahart Skype: Neridahartau