reviewing the quality of evidence a different act of remembering
Download
Skip this Video
Download Presentation
Reviewing the quality of evidence – a different act of remembering

Loading in 2 Seconds...

play fullscreen
1 / 12

Reviewing the quality of evidence – a different act of remembering - PowerPoint PPT Presentation


  • 93 Views
  • Uploaded on

Reviewing the quality of evidence – a different act of remembering. David Earle Tertiary Sector Performance Analysis Ministry of Education, New Zealand. Problem definition. Evaluation projects start out with high expectations Projects then encounter a number of ‘real life’ difficulties

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Reviewing the quality of evidence – a different act of remembering' - peter-perkins


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
reviewing the quality of evidence a different act of remembering

Reviewing the quality of evidence – a different act of remembering

David Earle

Tertiary Sector Performance Analysis

Ministry of Education, New Zealand

problem definition
Problem definition
  • Evaluation projects start out with high expectations
  • Projects then encounter a number of ‘real life’ difficulties
  • Results are disappointing and fail to convince
  • A review of literature reveals similar issues in previous projects
  • “If only we had known when we had started”
  • How often will we repeat past mistakes?
solution
Solution
  • Start by looking at other studies first
  • Even before evaluation design or framing is undertaken
  • To examine:
    • What designs or methods are possible?
    • What approaches have worked and not worked?
    • Which results have been useful and persuasive for end users?
  • Use the process to engage both researchers and end users in a discussion
situation 1 the evaluation literature
Situation 1: The evaluation literature
  • Adult literacy, language and numeracy research in New Zealand
  • Limited evaluative research in New Zealand
  • Several large scale projects have been completed in the UK and North America
  • Large number of case study and qualitative projects international
  • Large professional practice literature
  • Several substantive literature reviews and evidence syntheses
situation 2 the resources
Situation 2: The resources
  • Large new national data sets developing:
    • Literacy and numeracy for adults assessment tool
    • Education participation histories
    • Linked to employment outcomes (in future)
  • Need to complement with case study and longitudinal research
  • Limited resources spread across five organisations, with different interests and mandates
  • Need to build a conversation about what is quality, feasible, meaningful, important, useful etc.
approach scope of enquiry
Approach: Scope of enquiry
  • Not an evidence synthesis: focus on methodology, not on findings
  • Interested in low quality, failed studies as much as high quality, successful studies
  • Also interested in discussions around research projects
  • And development of research instruments and approaches
approach going about it
Approach: Going about it
  • Start with the broad literature reviews: often have good discussion sections
  • Evidence syntheses less useful as they bias towards specific methods and exclude exploratory work
  • Don’t need to start with a comprehensive database search (but may help)
  • Use free form note taking: don’t start templating references too early “Don’t know what you don’t know”
some early benefits
Some early benefits
  • Unveiling the rich debate of research and practice
    • Literacy and numeracy as social practice: not something that can always be measured on a scale
    • Range of background influences on learners that may be missed in ‘standard research’
  • Documenting recurring difficulties encountered in research designs
    • Falling response rates
    • Difficulties in establishing comparison groups
more early benefits
More early benefits
  • Finding examples of small scale innovation
    • Field trial of “non-academic outcomes”
  • Developing fuller range of research questions
    • Seeing what has already been examined and ‘proved’
    • Identifying recurrent “questions for further research”
    • Avoiding looking at things that don’t make a real difference
    • Getting beyond novelty of programme description
some early risks
Some early risks
  • Process is time consuming
  • Uncertain how to get wider, deep engagement beyond the authors: danger of each research project repeating the process
  • Field is extensive: difficult to cover all bases and all topics
discussion
Discussion
  • Other examples of this being taken as a systematic approach?
  • With emphasis on methodology rather than findings
further discussion
Further discussion

Twitter: datatoknow#method

ad