Readability Assessment of British Internet Information Resources on Diabetes Mellitus Targeting Laypersons

1. Readability Assessment of British Internet Information Resources onDiabetes Mellitus Targeting Laypersons Maged N. Kamel Boulos, Ph.D. School for Health, University of Bath M.N.K.Boulos@bath.ac.uk

2. Agenda • Introduction • Material and Methods • Results • Discussion and Recommendations • Conclusions and Useful Web Resources

3. Introduction

4. On Diabetes Mellitus • Diabetes mellitus, with its different clinical types, is an important public health condition in the UK and elsewhere, accounting for a sizeable percentage of the annual NHS expenditure. • The education of members of the public, patients and their carers is widely recognised as an essential component of chronic disease care and effective health promotion. • Appropriate education can minimise the risk of diabetes mellitus and its complications. • The Internet could be an extremely useful and cost-effective medium in this respect.

5. On Readability • Readability is an attempt to match the reading level of written material to the ‘reading with understanding’ level of the reader. • Readability (or understandability) is a frequently overlooked aspect of health information quality and accessibility. • For example, a recent evaluation by Thakurdesai et al. (2004) of the quality and contents of diabetes mellitus patient education on the Internet failed to test the reviewed sites for readability.

6. On Readability – Cont’d • Nearly half of all American adults or 90 million people have limited English and math skills (most are native-born English speakers), making it difficult for them to fully and safely understand and act upon health information (US Institute of Medicine, 2004). • Comparable proportions have been reported in the UK according to the Office for National Statistics. IOM Report Cover

7. Readability Testing Methods • More costly and time-consuming, but also more precise methods: • A comprehension test based on the document of interest administered to a group of readers of known reading ability. • Measuring readability by the judgment of a ‘literacy expert’. • Quick and rough approximations: • Using readability formulae: mathematical equations that estimate the reading level of a document based on the words that are used and the lengths of sentences. • Dozens of different readability formulae (e.g., Fry, SMOG and Flesch/Flesch-Kincaid formulae) have been developed over the years and many are implemented today in software.

8. Material and Methods

9. Sample Selection • We used Google (http://www.google.co.uk/ - UK pages) to locate British Internet information resources on diabetes mellitus targeting laypersons. • The ‘links’ sections of some of these resources were also explored to pick further UK sites. • In total, we selected 15 trustworthy English language sites/providers for inclusion in this study (all of UK provenance).

10. Readability Formulae and Software Used • We tested the readability of 20 representative pages from these 15 sites, providing consumer information on diabetes mellitus. • The full text from each of these pages was opened in Microsoft Word 2000 (any page navigation text was removed), and a readability level was generated using the Flesch Reading Ease score and Flesch-Kincaid Grade Level score.

11. Readability Statistics in MS Word Readability statistics in Microsoft Word for the text of http://www.diabetes.org.uk/infocentre/pubs/Whatc.doc The dialogue box on the left shows where to enable this function (‘Show readability statistics’) in Microsoft Word.

12. Calculation of Flesch Reading Ease and Flesch-Kincaid Grade Level scores

13. Transforming Results to Equivalent UK Reading Age and Computing Statistics • The reading level in US school grade (Flesch-Kincaid) was then transformed to the equivalent UK reading age. • For example, a Flesch-Kincaid Grade Level score of 8.0 equates to the UK reading age of an educated 13 year old person. • Mean and median readability scores of all sites/pages and other statistics were computed in Microsoft Excel 2000.

14. Results

15. Required Reading Age of Tested Material • The average and median UK reading ages of the sites/pages tested were 14.2 and 14.65 years of age respectively (range: 11-16.9 years). • These figures are well above the estimated reading age of the UK population in general (9 years or Year 4/5 at UK school).

16. Results – Cont’d • Only 7 pages (out of 20 tested, i.e., 35% of all tested pages) from 4 sites/providers (out of 15 included in this study, i.e., 26.66% of all sites/providers evaluated in this study) had a Flesch Reading Ease score above 60 (range: 60.4-73.3) and a Flesch-Kincaid Grade Level of 8.0 or lower (range: 6.0-8.0) corresponding to a UK reading age of 11-13 years. • The remaining sites/pages in this study scored worse (Flesch Reading Ease score range: 31.6-57.1 and Flesch-Kincaid Grade Level range: 9.2-11.9), with a corresponding UK reading age of 14.2-16.9 years.

17. Detailed Results

18. And the Winner Is… • NHS Direct Online health encyclopaedia (http://www.nhsdirect.nhs.uk/)scored among the worst sites in this study, while the patients’ section of BestTreatments (BMJ Publishing Group/NHS Direct -http://www.besttreatments.co.uk/)came out as the best, and most readable site in our study. • As expected, the two readability scoring formulae used in our study were found to be very highly correlated (a high negative correlation of -0.92124, measured using Microsoft Excel CORREL function; when Flesch Reading Ease increases, Flesch-Kincaid Grade Level decreases and vice versa).

19. Discussion and Recommendations

20. How Does This Study Compare With Previous Studies? • Our results echo those of similar studies conducted over the past few years, and point to a chronic and often forgotten information quality problem. • Berland et al. (2001) reviewed 25 English and Spanish-language health Web sites and concluded that “much of the health information available on the Internet is beyond the comprehension of many consumers”. • Kusec et al. (2003) studied the readability levels of 99 diabetes-related Web sites displaying the HONcode logo (http://www.hon.ch/) and found that 86.9% of the tested material would be too difficult to read for the average adult population. • Chestnutt (2004) evaluated the readability of 100 patient-related Internet sites presenting information on common oral pathologies and reported a mean UK reading age of the evaluated sites of 10-11 years, which is higher than the UK national reading age.

21. Limitations of Readability Formulae • There are many factors that affect readability and understandability of online health information, which are not measured by readability formulae. • Rudolf Franz Flesch, who created the Flesch readability formula, warned that “Some readers, I am afraid, will expect a magic formula for good writing and will be disappointed with my simple yardstick. Others, with a passion for accuracy, will wallow in the little rules and computations but lose sight of the principles of plain English. What I hope for are readers who won't take the formula too seriously and won't expect from it more than a rough estimate”.

22. But They Remain Better Than Nothing... • Nevertheless, we recommend that any health information quality benchmarking scheme or checklist in use should include (or be supplemented with) some explicit testing for content readability. • Readability testing results could be also displayed on the tested pages to inform readers (‘a readability seal’).

23. Making Online Consumer Health Information Easier to Understand • Ensure that written material is pitched at a level appropriate to the intended lay audience (aim to match the general reading level of the UK population: nine years or Year 4/5 at UK school). • Write shorter sentences and paragraphs. • Use the active voice. • Avoidjargon and use simpler, common words. • A picture is worth thousand words: include real world examples and easy-to-understand illustrations (where applicable).

24. Practical Example: (Re-)writing for Readability Quoted fromhttp://www.pfizerhealthliteracy.com/improving_empathy.html Original text - Flesch Reading Ease 14.5:A living will is a written declaration directing your doctor to provide, withhold or withdraw lifesaving procedures should you be diagnosed as having a terminal condition or exist in a vegetative state and you lack the capacity to make the decision.(Source: Planning for Incapacity, Legal Council for the Elderly) Revised text in an easy-to-read format - Flesch Reading Ease 68.1 (^ = better):A living will is a paper you sign telling your doctor whether you want treatments that could keep you alive if you have a terminal disease or are in a coma. This lets your doctor know how you want to be treated if you are not able to make the decision yourself.

25. Making Online Consumer Health Information Easier to Understand – Cont’d • Properly structure and organise information into sections and subsections with meaningful headings and subheadings. • Use bulleted lists (where applicable). • An inverted pyramid writing style is sometimes also helpful (start the article by telling the reader the conclusion/action(s) to take, followed by the most important supporting information, and end by giving the background).

26. Making Online Consumer Health Information Easier to Understand – Cont’d • Consider the specific cultural needs and socio-economic levels of different ethnic groups. For example, health education material should describe alternative options in case of lack of economic resources, when referring to processes that imply large expenses, e.g., some costly diabetic diet recipes. • Also pay attention to the visual style and presentation of online health information material, including issues like text/background colour and font type and size. These are equally important to improve information accessibility.

27. Consider Other Means of Delivery • However, even the most readable documents will remain difficult to fully and properly understand for a sizeable proportion of the population. • For this reason, in addition to written online material, health information providers should also consider other means of delivering health education and promotion programmes like face-to-face education and plain English videos, so that no one is left behind. • The videos could be also made available online and provided in additional languages spoken by ethnic minorities/immigrants.

28. Conclusions and Useful Web Resources

29. Not All Patients Are Experts! • During recent years, there has been a growing trend of treating medical knowledge as a single blob that is relevant to both professionals and laypersons, and so should be made accessible to all groups without any distinction. • Supporters of this trend believe that patients should be empowered and given more information and control of their conditions. Indeed, laypersons sometimes show more knowledge and understanding of their own conditions than their treating doctors do. • In spite of that, it seems sensible to continue writing different information sets for the healthcare professional, the highly educated and the average layperson, and to label online health information according to its intended primary audience.

30. Conclusions – Cont’d • Besides the fundamental requirement to check the medical correctness or soundness, completeness, accuracy, and currency of health information targeting laypersons, online health information providers also need to ensure that their written material is pitched at a level appropriate to their intended lay audience. • This is an equally important requirement as it affects the final utility and reliability of published online material. • Public and patient health information that is difficult to understand or liable to misunderstanding by the lay consumer could result in serious consequences.

31. Useful Web Resources • Pfizer Clear Health Communication - http://www.pfizerhealthliteracy.com/and their handbook -http://www.pfizerhealthliteracy.com/4548_Health_literacy_all.pdf • The Center for Health Care Strategies’ Fact Sheets on Health Literacy -http://www.chcs.org/