1 / 19

RACcER: Re-use and Archiving of Complex Community Based Evaluation Research

RACcER: Re-use and Archiving of Complex Community Based Evaluation Research. Irish Qualitative Data Archive Tallaght West Childhood Development Initiative. Irish Qualitative Data Archive. Funded by the Irish Government under the Programme for Research in Third Level Institutions (Cycle 4)

materia
Download Presentation

RACcER: Re-use and Archiving of Complex Community Based Evaluation Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. RACcER: Re-use and Archiving of Complex Community Based Evaluation Research Irish Qualitative Data Archive Tallaght West Childhood Development Initiative

  2. Irish Qualitative Data Archive • Funded by the Irish Government under the Programme for Research in Third Level Institutions (Cycle 4) • Several major datasets being archived, including – • Growing up in Ireland study • Life Histories and Social Change in Ireland in 20th Century Ireland • But... • Limitedawareness/understanding of qualitative archiving in Irish research community • Lack of literature about challenges of archiving qualitative data generated in evaluation research • Re-use & Archiving of Complex Community-based Evaluation Research (RACcER) project explores issues specific to archiving community-based evaluation data, also aims to raise awareness throughout research community

  3. RACcER • RACcER co-funded by IRCHSS and CDI: demonstrator project for qualitative archiving • Objectives: • Scope concerns, issues and requirements in qualitative data archiving through interviews • Explore issues and challenges for researchers, participants, funders, potential data users • Test feasibility of archiving evaluation data • Develop protocols and procedures for archiving evaluation data • Research • Interviews with 30+ individuals from policy and research community, including those involved in TWCDI evaluation • Evaluation researchers were all university based

  4. Tallaght West Childhood Development Initiative (CDI) • 1 of 3 Prevention and Early Intervention Programmes jointly funded by government & philanthropy • Aims to improve the education and well-being outcomes for children and families in historically disadvantaged areas of Tallaght West • 5 constituent projects: early years, literacy, pro social behaviour, healthy schools, community safety • Major emphasis on evaluation • Mainly quantitative approach to evaluation of RCT interventions, but qualitative process data collected for all constituent projects and overall evaluation

  5. Findings – qualitative data gathered in evaluations • Range of qualitative material collected • One-to-one interviews, discussion groups, meeting minutes, site observations, project log books • Variety of participant groups • Service providers, practitioners, school, police, health service personnel, service users (children and parents), community reps • Most material recorded, anonymised and transcribed

  6. Findings – archiving policy • Evaluators’ research centres didn’t generally have archiving policies for qualitative material • Range of experience of what happens to qualitative data after project • Destroyed; stored privately but not used; returned to commissioner; archived internally • No evidence of researchers sharing datasets informally • Default ethics position is to destroy after 5 years • In some cases 10 years

  7. Findings – archiving experience • No evidence of formal archiving • Limited experience of accessing qualitative archives, • One example for use in teaching • Some had archived quantitative material • Some had accessed quantitative archives • Range of views on resource implications of prepare qualitative material for archiving

  8. Findings – issues with archiving • Difficulties of anonymising in small community • Level of anonymisation would reduce re-use value • Perception of limited reuse value of material collected for evaluation purposes • Secondary analysts wouldn’t understand the nuances of evaluation context • What’s the added value of raw material vs. final report? • Lack of awareness/understanding/trust in newly established archive • Compromises fully informed consent • Viability of archive, level of resources queried • Reluctance to ‘hand over’ data to be governed by unknown institution

  9. Findings – issues(2) • Consent not sought • Unethical to archive without consent • Retrospective consent not feasible • ‘Ethically dubious’ to archive documents like meeting minutes, no consent • Constraints of ethics committees limit archiving • Loss of control means consent can’t be fully informed • Prospect of archiving reducing quality of the research a major concern • Reduce willingness to participate in research • Constrain what participants willing to share • “There wasn’t a hope in hell of anyone speaking to us unless there was anonymity.” • Archiving seen to equate with loss of anonymity

  10. Findings – issues(3) • Wide perception of fundamental difference between evaluation research and broader topics • Necessarily ‘political’ • At its core it is a judgement on the management and delivery of services • Interview concerns 3rd parties, consent not sought of them • Services, jobs, funding are in question, highly sensitive • Usually geographically bounded, problems with anonymisation

  11. Findings – issues(4) • Lack of clarity on who can access data and how it could be used generates fear and reduces inclination to archive • Question of what constitutes a ‘bona fide researcher’? – • How would bona fides be established? What assurances? • Issue of policy researchers’ & communities’ eligibility to access • Concern over data being used out of context, ‘twisted’, sensationalising populations, journalists gaining access • Archiving seen as a breach of trust between the researcher and researched

  12. Findings – issues(5) • Researchers concerned about being opened up to scrutiny • Sense of archiving being used to open up research to scrutiny makes researchers feel they aren’t trusted • Potential for commissioners/others to access the archive to undermine research when they don’t like what it has said • Influence of that on evaluation research process “And if they’re going to check my work I might as well write what they want to hear.” • PhD/masters students using archived material could reduce valuable learning opportunities gained from fieldwork • Resource implication of archiving • Benefits not worth gain if funders aren’t supporting it

  13. Findings – advantages • Potential to explore alternative research questions • Macro level implementation questions across PEIP sites • Language used by different groups in schools • Impact of setting on parents’ willingness to disclose information • Intangible factors in project success such as charisma • Contributing to cumulative knowledge of implementation of children’s services • Facilitate comparative analysis over time or place • Although some qual researchers dispute extent to which comparison is appropriate

  14. Findings – advantages (2) • A valuable teaching resource • But disputed by some as being unethical • Increase the standing of qualitative research if it is open to external examination, improved validity and reliability • But concerns about how it could be used in this way • Contribute to corporate memory of commissioning organisation • But is this the purpose of archiving?

  15. Findings - risks • Reputational risk for organisations involved at all levels • Particularly if initiatives haven’t been successful • Equally if commissioner is unhappy with research findings • Negative publicity for researched community • User agreement could be breached • Anonymity could be compromised – • Longer term implications for researchers accessing certain communities • But... • Need to consider likelihood of risk occurring and how serious it would be if it happened

  16. Findings – mitigating risks • Strict controls on user access • Ethics approval • Sign-off of original researcher on publications • Archive to facilitate collaboration between researchers • As an explicit function of the archive, builds trust • Hierarchy of access, tailored depositor and user agreements • Some accept that time delay on access reduces risk • Others say time doesn’t make a difference, especially in with research on children • Time delays reduce reuse-ability in policy context • ‘Sunset clause’ may also be useful in some cases

  17. Other project issues • Data ownership issue in evaluation context was very sensitive • Massive underlying issue in this project – part of the genesis of RACcER • Commissioners claim legal ownership, evaluation teams dispute this, some suggest participants have moral ownership • Underestimated importance of anonymity to research participants • Implications for archiving & reporting–anonymisation burden, higher level unit of analysis • Some participants refused archiving

  18. Anticipated project outputs • Report on RACcER research  • Development and dissemination of best practice guidance • Response to concerns identified in research • Workshop with research participants • Raise awareness about archive and provide support • Archiving of CDI internal material • Archiving of RACcER interviews  

  19. RACcER Team • Irish Qualitative Data Archive • Jane Gray • Julius Komolafe • Hazel O’Byrne • Aileen O’Carroll • Tallaght West CDI • Tara Murphy

More Related