1 / 18

Improving the quality of responses to health search queries by evidence-based context

Improving the quality of responses to health search queries by evidence-based context. Peter Bailey Project Leader, Search and Delivery, ICT Centre 10 August 2007. Health searching – what will benefit me most?. Accuracy – relevance Accountability – quality Anonymity – privacy

brina
Download Presentation

Improving the quality of responses to health search queries by evidence-based context

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improving the quality of responses to health search queries by evidence-based context Peter Bailey Project Leader, Search and Delivery, ICT Centre 10 August 2007

  2. Health searching – what will benefit me most? • Accuracy – relevance • Accountability – quality • Anonymity – privacy • Current “generic” web search faces many challenges CSIRO Improving the quality of responses to health search queries by evidence-based context

  3. Accuracy – relevance • Respond to possible information needs with • Highly relevant information • Assistance in refining need • Meet this goal more successfully by better use of contextual information about the person • Their task requirements, e.g. • Research • Self diagnosis • Checking medical recommendations • Their personal circumstances, e.g. • Location, age, sex, … • Availability of treatments • Current health status CSIRO Improving the quality of responses to health search queries by evidence-based context

  4. Health search – depression CSIRO Improving the quality of responses to health search queries by evidence-based context

  5. Health search – obesity CSIRO Improving the quality of responses to health search queries by evidence-based context

  6. Health search – diet CSIRO Improving the quality of responses to health search queries by evidence-based context

  7. Health search – fat CSIRO Improving the quality of responses to health search queries by evidence-based context

  8. Accountability – quality • Low-quality Web information • Lots of it • Serious consequences if misleading advice followed • How to get rid of it from results? • Evidence-based medicine information • Treatments recommended according to strength of scientific evidence • Multiple randomised controlled trials • How to maximise it in results? CSIRO Improving the quality of responses to health search queries by evidence-based context

  9. Health search – depression treatments CSIRO Improving the quality of responses to health search queries by evidence-based context

  10. Automated quality assessment (AQA) • Relevance and quality queries were learned • An extension and application of relevance feedback techniques • Selection of indicative query terms using Term Selection Values (TSVs) • Single word terms and two-word phrases; highest 20 TSVs selected (for relevance) and 29 words and 20 phrases (for quality) • Numerical weights assigned using Robertson-Spärk Jones formula • Training documents used with background pool of non-relevant, or relevant but not high quality documents • Combination of relevance and quality scores for websites • Scoring algorithms normalised, and then scaled to be comparable with human rating scales CSIRO Improving the quality of responses to health search queries by evidence-based context

  11. AQA – correlations • AQA site scores were correlated 0.85 (p < .001) with human expert judgements of site quality • Linear correlation, with additional hierarchical multiple regression analysis • Compares almost with inter-rater reliability • Small and non-significant correlation of PageRank with expert judgements (r=0.23, p=.22) • PageRank calculated from Google Toolbar’s value (0-10) • Excluding PageRank site scores of 0, a correlation exists of 0.61 (p<.002) • Still significantly lower than the association between AQA and evidence-based quality scores • Site score then used as the AQA metric for result filtering, re-ranking, or crawler selection policy CSIRO Improving the quality of responses to health search queries by evidence-based context

  12. AQA – in action CSIRO Improving the quality of responses to health search queries by evidence-based context

  13. AQA – performance and analysis • Measurements are on a site basis, not a page basis • Judging took 6 months • AQA-based search engines outperformed others • Rank-averaging outperformed on quality • Filtering outperformed on relevance • Quality focused crawling is more viable than manual definition CSIRO Improving the quality of responses to health search queries by evidence-based context

  14. AQA – for quality focused crawls • Poor quality pages found early in crawl • Continued growth in high quality pages as crawl progresses • Relies on unbroken link chains between seedlist and relevant high quality content • Generic search engines with high crawl coverage have an advantage • Opportunity however for dedicated vertical search CSIRO Improving the quality of responses to health search queries by evidence-based context

  15. Anonymity – privacy • How do I preserve my privacy when interacting with a search engine? • Health search matters are some of the most deeply personal • No wish to hand this out to anyone • See AOL query logs release (Aug 2006), and recent announced changes by Google, AOL, … on information retention • Imagine if spam email started drawing on your searches • Also advertisements are not always correlated with quality health advice • Spam and other adversarial techniques need combating CSIRO Improving the quality of responses to health search queries by evidence-based context

  16. Future work • Australia (and other countries) faces a growing health challenge of obesity • Associated with poor health in later life, such as type 2 diabetes, increased heart disease, liver disorder, … • Currently applying AQA methods to health topic of obesity • Expert classification of obesity information from web sites by CMHR • Need to learn relevance and quality queries • Conduct comparative experiments • Can AQA apply generically? • How to obtain health topic specific training material for learning AQA CSIRO Improving the quality of responses to health search queries by evidence-based context

  17. Acknowledgements • Pictures licensed under Creative Commons from Flickr • Chinese hospital - http://www.flickr.com/photos/shapeshift/ • Baby footprint - http://www.flickr.com/photos/hamed/ • Three girls - http://www.flickr.com/photos/malingering/ • Darts - http://www.flickr.com/photos/ramk13/ • Scrolls - http://www.flickr.com/people/felix42/ • Laptop browser - http://www.flickr.com/photos/shapeshift/ • Lego heads - http://www.flickr.com/photos/richard_am/ • Work being carried out in collaboration with: • Kathy Griffiths (ANU), David Hawking (CSIRO), Ramesh Sankaranarayana (ANU), Thanh Tang (ANU) • Funding by: • CSIRO ICT Centre • Australian National University • Microsoft Research Asia CSIRO Improving the quality of responses to health search queries by evidence-based context

  18. ICT Centre, Search and Delivery Peter Bailey Project leader Phone: +61 2 6216 7055 Email: Peter.Bailey@csiro.au Web: www.csiro.au/ict Thank you Contact UsPhone: 1300 363 400 or +61 3 9545 2176Email: Enquiries@csiro.au Web: www.csiro.au

More Related