Reviewing the quality of evidence a different act of remembering
This presentation is the property of its rightful owner.
Sponsored Links
1 / 12

Reviewing the quality of evidence – a different act of remembering PowerPoint PPT Presentation


  • 58 Views
  • Uploaded on
  • Presentation posted in: General

Reviewing the quality of evidence – a different act of remembering. David Earle Tertiary Sector Performance Analysis Ministry of Education, New Zealand. Problem definition. Evaluation projects start out with high expectations Projects then encounter a number of ‘real life’ difficulties

Download Presentation

Reviewing the quality of evidence – a different act of remembering

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Reviewing the quality of evidence a different act of remembering

Reviewing the quality of evidence – a different act of remembering

David Earle

Tertiary Sector Performance Analysis

Ministry of Education, New Zealand


Problem definition

Problem definition

  • Evaluation projects start out with high expectations

  • Projects then encounter a number of ‘real life’ difficulties

  • Results are disappointing and fail to convince

  • A review of literature reveals similar issues in previous projects

  • “If only we had known when we had started”

  • How often will we repeat past mistakes?


Solution

Solution

  • Start by looking at other studies first

  • Even before evaluation design or framing is undertaken

  • To examine:

    • What designs or methods are possible?

    • What approaches have worked and not worked?

    • Which results have been useful and persuasive for end users?

  • Use the process to engage both researchers and end users in a discussion


Situation 1 the evaluation literature

Situation 1: The evaluation literature

  • Adult literacy, language and numeracy research in New Zealand

  • Limited evaluative research in New Zealand

  • Several large scale projects have been completed in the UK and North America

  • Large number of case study and qualitative projects international

  • Large professional practice literature

  • Several substantive literature reviews and evidence syntheses


Situation 2 the resources

Situation 2: The resources

  • Large new national data sets developing:

    • Literacy and numeracy for adults assessment tool

    • Education participation histories

    • Linked to employment outcomes (in future)

  • Need to complement with case study and longitudinal research

  • Limited resources spread across five organisations, with different interests and mandates

  • Need to build a conversation about what is quality, feasible, meaningful, important, useful etc.


Approach scope of enquiry

Approach: Scope of enquiry

  • Not an evidence synthesis: focus on methodology, not on findings

  • Interested in low quality, failed studies as much as high quality, successful studies

  • Also interested in discussions around research projects

  • And development of research instruments and approaches


Approach going about it

Approach: Going about it

  • Start with the broad literature reviews: often have good discussion sections

  • Evidence syntheses less useful as they bias towards specific methods and exclude exploratory work

  • Don’t need to start with a comprehensive database search (but may help)

  • Use free form note taking: don’t start templating references too early “Don’t know what you don’t know”


Some early benefits

Some early benefits

  • Unveiling the rich debate of research and practice

    • Literacy and numeracy as social practice: not something that can always be measured on a scale

    • Range of background influences on learners that may be missed in ‘standard research’

  • Documenting recurring difficulties encountered in research designs

    • Falling response rates

    • Difficulties in establishing comparison groups


More early benefits

More early benefits

  • Finding examples of small scale innovation

    • Field trial of “non-academic outcomes”

  • Developing fuller range of research questions

    • Seeing what has already been examined and ‘proved’

    • Identifying recurrent “questions for further research”

    • Avoiding looking at things that don’t make a real difference

    • Getting beyond novelty of programme description


Some early risks

Some early risks

  • Process is time consuming

  • Uncertain how to get wider, deep engagement beyond the authors: danger of each research project repeating the process

  • Field is extensive: difficult to cover all bases and all topics


Discussion

Discussion

  • Other examples of this being taken as a systematic approach?

  • With emphasis on methodology rather than findings


Further discussion

Further discussion

  • [email protected]

    Twitter: datatoknow#method


  • Login