Qualitative Research Study on Environmental Health and Toxicology

Qualitative Research StudyonEnvironmental Health and Toxicology Prepared for: April 12, 2006

Background • Study Objectives • Methodology: Online Focus Group • Summary Conclusions • Overall Findings: Environmental Health & Toxicology • Recommendations • Appendix

Background • As part of an ongoing qualitative evaluation of NLM websites, the National Library of Medicine conducted TWO online focus groups on Environmental Health and Toxicology – a portal to databases and resources to help professionals and laypeople in the environmental health and toxicology community. • The primary objective of the study was to understand the value, usefulness, strengths, and weaknesses of the Environmental Health and Toxicology website for potential users of the site -- both professional and non-professional. • The online focus groups on the Environmental Health and Toxicology site were conducted on April 6, 2006.

Study Objectives • The main objective of this qualitative study on Environmental Health and Toxicologywas to evaluate the value and usefulness to current and potential users. Additional objectives included: • Gauging the site's strengths and weaknesses, in terms of content, navigation, and overall structure • Understanding how users currently use and intend to use the site • Getting reactions to acronyms and other terminology used on the site Target respondents. Screening parameters for respondents were as follows: • Mix of current and potential users • Group 1 was comprised of professionals employed in the field of environmental health and toxicology as physicians, academics (educators, researchers, etc.), nurses, and librarians. • Group 2 was comprised of concerned citizens -- those with a strong concern about chemicals in the environment and interest and/or participation in activities related to conservation and environmental health. • Group 1 was recruited from a listserv subscription within the environmental health and toxicology community; group 2 was screened and recruited via telephone using a recruiting agency. • All were asked to spend approximately 20 minutes viewing/navigating the site and completing a series of tasks prior to the discussion.

Methodology: Online Focus Group* • Testing consisted of TWO (2) 90-minute online focus groups. • Respondents were recruited via listserv subscription (Professionals) and telephone (Concerned Citizens). • Group 1 (Professionals) consisted of 11 participants, and Group 2 (Concerned Citizens) consisted of 12 participants. (See Appendix.) • The group was asked to visit the Environmental Health and Toxicology site prior to the session. They were also asked to complete a series of tasks using the site. • Participants in the "Professionals" group received an incentive payment of $45 (though two participants waived payment), and participants in the "Concerned Citizens" group received a $75 incentive payment. • * The online focus group represents a qualitative methodology used for the purposes of ideation, brainstorming, and evaluation. Qualitative methodologies are based on a small sample size, and the findings are intended to be directional only, not projectable to the larger population.

Summary Conclusions • Impressions of Environmental Health and Toxicology • Reactions to the Environmental Health and Toxicology website from both Professionals and Concerned Citizens were enthusiastically positive. • Respondents offered several constructive critiques of specific areas of the site, but overall, it was perceived as an extremely useful, handy, valuable tool for the environmental health and toxicology community -- for both the public and professional sectors. • Respondents liked the idea of having a central portal to use as a "starting point" for all of their searches on chemical toxicity and related topics. • Very few had difficulty navigating, though there were some areas that seemed overwhelming to respondents, in terms of the sheer amount of content. • Minor obstacles emerged as areas for improvement that could be addressed with tweaks to the site. For instance, there was some confusion as to the purpose of three search mechanisms on one screen -- for SIS, TOXNET, and NLM. • The use of acronyms presented confusion to some respondents -- particularly in the Citizens group (who were less familiar with what they represented) -- but this was seen as easily addressed with mouse rollovers or on-screen definitions.

Summary Conclusions (cont’d.) • Strengths and Weaknesses: Environmental Health and Toxicology • Strengths of the Environmental Health and Toxicology site were its logical and "clean" layout, ease of use, and comprehensive coverage of information. • Weaknesses of the site included the (perceived) excessive use of acronyms (without obvious definition) and the presence of multiple search tools. • Suggestions for change included expanding the FAQ section; adding categories such as "community volunteers," "parents," "kids," and "risk assessors" to the "Especially For" section; and adding on-screen definitions of the mouse rollovers for the acronyms. • Several suggestions emerged for expanding some areas (such as "Especially For," as mentioned above, and "Reference Tools") and shortening others (e.g., "More to Explore"), but overall, the site was perceived as extremely useful. • The concept of online tutorials or videos was met with a mixed reaction. Most liked the idea of online tutorials, provided there was no cost, but the idea of videos on-screen yielded concerns about download time and other associated factors (e.g., access speed).

Summary Conclusions (cont’d.) • Environmental Health and Toxicology • Respondents expect to use the Environmental Health and Toxicology site frequently, and in many cases have already referred colleagues and friends to the site. • Overwhelmingly, the issue of data reliability was high (trustworthy and reliable), though a select few in the Citizens group expressed concern that a site associated with the government may have a political bias or "policy-based" censorship. • Respondents would like to learn more about NLM resources via e-mail, conferences, local community organizations, and professional organizations. • In contrast to other resources in the field of environmental health and toxicology, however, the site was seen as head-and-shoulders outstanding. • Overall, the site does a great job of providing a comprehensive, inclusive starting point for both professionals and laypeople seeking in-depth information on environmental health, toxicology, and related topics. Following are detailed findings from the testing on Environmental Health and Toxicology.

Overall Findings: Environmental Health and Toxicology • Current information gathering • Most respondents use the Internet to find information on environmental health and toxicology. • Current patterns of information-gathering on environmental health differ between the Professionals and the Concerned Citizens. Professionals reported going to NLM sites and professional organization sites (e.g., NIOSH, OSHA, EPA, WHO), while Citizens are more likely to search for information via Google or another Internet-based search engine. • Some also use the CDC, PubMed, and WebMD. • Users in the Professionals group reported that they characteristically seek information on health effects of chemicals, peer-reviewed studies and scholarly research, and toxicity data on household and other products. • Those in the Concerned Citizens group are more interested in a broad range of topics that impact their own situations or their daily lives. Examples included floor cleaners, medicines and prescriptions, clean water issues, automotive oil, radiation and chemotherapy.

Overall Findings: Environmental Health and Toxicology • Current information gathering (cont'd.) • Respondents in both groups also were asked if they seek information on specific topics. • The majority of professionals but few citizens said that they seek information on nitrogen compounds, solvents, gases/vapors, plastics and rubber. In the Citizens group, some pointed out that they had searched for information on hybrid cars and garage chemicals. • Most professionals reported that they do seek information on pesticides, but among Concerned Citizens, there was less interest in this topic. • Hazardous household products was reported as a common search topic among Concerned Citizens, and some in the Professionals group also seek information in this area. • "Occupational safety" was not high on the list of search topics in either group, but "drinking water," on the other hand, was an increasingly major search topic across the board.

Overall Findings: Environmental Health and Toxicology • Common Challenges in Gathering Information • The types of challenges commonly encountered in seeking environmental health and toxicity information involves recency of data, "wading through" the sheer breadth of information (which can be overwhelming, particularly to the layperson), and -- for both groups -- finding a "one-stop" site for this type of information. • For some, the specificity of the information made the information seemed largely geared toward the Professional user. “The databases are so specific that it's often a challenge to know which one to search for the exact data.” (Megan, Professional) “sometimes there’s too much info - where to start?” (Lori, Citizen) “The lay person's expression of toxicity and risk are very different from the professional community's use of those words. They are more broad, ours more specific.” (Marcus, Professional)

Overall Findings: Environmental Health and Toxicology • Common Challenges in Gathering Information (cont'd.) • Similarly, several in the Citizens group explained that the terminology is not fully accessible for the layperson -- that it tends to be "too technical" and requires "translation." • Additionally, for some in the Citizens group (possibly due to lower recognition of the sources), there was some concern as to the reliability of the information. “The lay person's expression of toxicity and risk are very different from the professional community's use of those words. They are more broad, ours more specific.” (Marcus, Professional) “Usually the info is either too general to be useful, or so technical that's it hard to interpret. If it's anything chemically related, I ask my father the chemist!” (Shalom, Citizen) “reliability of info - can it be trusted?” (Laura, Citizen) “… sometimes the commercial websites have the most information, and the challenge … is to read through the lines because … they are trying to sell their products!” (Lorraine, Citizen) “Challenges include sorting through the massive amount of literature. Who is right, wrong, solid research or a professor publishing something for tenure. I do not know who I can trust if a private site.” (Mark, Citizen)

Overall Findings: Environmental Health and Toxicology • Addressing these Challenges • Respondents reported that they typically address these challenges by limiting search terms, reorganizing the information so they can "digest" it, and checking with peers and colleagues. • On the other hand, those in the Citizens group reported addressing challenges by checking information for consistency against other sources and occasionally asking others. “resolving challenges -- time, brainbusting, talking it out with respected peers, trying to find a new perspective/out of the box angle.” (Marcus, Professional) “When searching toxline, for example, I use a lot of very specific limiting terms.” (Nancy, Professional)

Overall Findings: Environmental Health and Toxicology • Impressions of Environmental Health and Toxicology • All of the respondents in the Professionals group but few of those in the Concerned Citizens group had heard of or visited either Environmental Health and Toxicology OR TOXNET before. • Regardless of past familiarity or visitation, however, impressions were overwhelmingly positive. • Several respondents in the Professionals group had used the site for information on poisoning, chemical exposure levels, and chemical structures. Several were also familiar with TOXMAP and Tox Town. "I had been there. I am familiar with a host of NLM sites and appreciated the similarities in layout.” (Megan, Professional) "My overall impressions were very POSITIVE. I like the fact that this is like the encyclopedia for environmental health.” (Shyam, Citizen) "This is very comprehensive and specific to particular concerns. It's very plain and surprisingly easy to use.” (Wendy, Citizen)

Overall Findings: Environmental Health and Toxicology • Strengths and Weaknesses • Some of the most positive aspects of the Environmental Health and Toxicology site were its clean layout and ease of use. • Respondents also felt that it was a good "starting point" for encyclopedic information on chemicals and toxicology and found it to be very useful. "Overall, homepage pulls together divergent but related topics and displays them in a clean layout. Provides quick access to literature citations and 'ready-reference' data.” (Heather, Professional) “Info on specific chemical hazards. Household was good.” (Linda, Citizen) "I think there are so many uses for this site. I will certainly tap into it for any personal concerns I have about toxins and I think it will be a great education resource, too.” (Juliana, Citizen) "I had no dislike when I visited it.” (Shyam, Citizen)

Overall Findings: Environmental Health and Toxicology • Strengths and Weaknesses (cont'd.) • Professional respondents encountered an obstacle of missing some of the information below the fold of the screen (requiring scrolling), feeling that the information was buried in layers accessible only after multiple clicks. • In the Citizens group, some complained of inaccessible terminology, FAQs that were insufficient for their needs, and an overwhelming amount of information. "The top section of the homepage emphasizes NLM resources. It takes several clicks to get to non-NLM resources.” (Nancy, Professional) "A lot is ‘below the fold’ on the screen when displayed with medium sized font. Layered headers at top take up a lot of real estate.” (Heather, Professional) "I wished the primary info that came up was less compound bound inasmuch as I'm a layperson.” (Nancy, Citizen) "I was not crazy about the FAQs-no one ever seems to answer my questions! … I always feel like the questions I have are not on the list and I have to email them to someone-by that time, I have found it on my own.” (Juliana, Citizen) "Tremendous resource, but a little overwhelming to be presented with everything all at once.” (Shalom, Citizen)

Overall Findings: Environmental Health and Toxicology • Strengths and Weaknesses (cont'd.) • Also in the Citizens group, one respondent had difficulty navigating (describing it as having "too much information"). • Also in the minority, one respondent in the Citizens group who visited Tox Town disliked the youthful orientation of the design, questioning for whom it was intended. "The site was hard to navigate and not very user friendly … There is too much information to filter through. The main page could be a link to different, separate home pages, each specializing in a topic instead of duplication within multiple categories.” (Mark, Citizen) “wasn't thrilled with the neighborhood picture - seemed a bit juvenile - are you targeting kids as an audience?” (Lori, Citizen) “I looked at Tox Town but was turned off by the animation.” (Lorraine, Citizen) "Tox Town was very confusing. I was looking for a section to put my zip code or city in.” (Robert, Citizen)

Overall Findings: Environmental Health and Toxicology • Comparison to Other Resources • The majority of respondents felt that the Environmental Health and Toxicology site was superior to other similar resources on the Internet. In contrast to TOXNET, it is perceived as more comprehensive. • Some of the best things about it included its consolidation of information and the ability to search multiple databases via one portal. "It is better, and easier for me to use … I like it very well as it is.” (Jim, Citizen) "Better. I really like having the multiple entry points -- both topical and audience.” (Megan, Professional) "The EHT site may be more inclusive, covering more audience.” (Christopher, Professional) "consolidation of all this info and connections at one homebase to search from and to be able to search all tox-related databases at once.” (SueAnn, Professional) "It actually contains a significant amount of the sites that I use--so it is a single portal--no extra typing.” (Lawrence, Professional)

Overall Findings: Environmental Health and Toxicology • Suggestions for Change • Suggestions for change included expanding the FAQ, adding mouseovers to the database names, and adding maps and "entrances" for the non-professional user. "I agree about the FAQ's and there should be different colours or texts to distinguish the different areas because your eyes don't know where they're moving from and to seemingly.” (Wendy, Citizen) "Move toxnet faq higher on screen. add mouse over description of what is behind items.” (Heather, Professional) "Maybe a series of relational or cognitive maps to navigate the site.” (Marcus, Professional) "Once I've gotten a few levels down, I tend to ‘get lost.’ I'd like to see every page have a map of the site.” (Nancy, Professional) “Any way to have mouseovers on database names to tell what's in each database (on blue toxnet bar on right side)?” (Megan, Professional) "I would add an 'Entrance' for lay persons, i.e., non chemists.” (Nancy, Citizen)

Overall Findings: Environmental Health and Toxicology • Suggestions for Change (cont'd.) • Additionally, there were suggestions for reorganizing the information in a way that emphasizes PURPOSE or REASON for the search. • One user also suggested including training tools, such as downloads, and more online tutorials. "Something to direct me at a high level *first* -- e.g., are you looking for information about chemicals, are you looking for material for teaching a class -- those are so different, they shouldn't be mashed together on the same page.” (Shalom, Citizen) “It would be helpful to have a link to health and safety training materials and resources, e.g., powerpoint downloads.” (Ron, Professional)

Overall Findings: Environmental Health and Toxicology • Target Audience • Most respondents in the Professionals group felt that the site was designed for a professional audience, while those in the Concerned Citizens group felt that it was geared toward both professionals and the general public (with a stronger orientation toward the savvy professional). • One respondent, however, noted that the site requires a certain amount of technical savvy to use. "bench scientists, regulatory scientists, at least." (Christopher, Professional) “Practitioners in the field--but ToxTown is more for ‘students.’" (Lawrence, Professional) “I think it's good for anyone, but will probably be used more by scientists and health professionals." (Nancy, Professional) “Professionals in the environmental health field. That was my first impression - can be used by anyone after looking at it." (Laura, Citizen) "Professionals and laypersons doing research." (Lorraine, Citizen) “Seems like it's trying to meet the needs of everybody, but assuming pretty sophisticated users / technically savvy.” (Shalom, Citizen)

Overall Findings: Environmental Health and Toxicology • The Use of Acronyms and its Impact on Perception of Target Audience • Specific to TOXNET, some respondents in the Concerned Citizens group felt that the site was not designed for the "average joe." This finding emerged from users' lack of familiarity with the acronyms used on the site. While the rollovers made a minor difference, non-professional users prefer "at-a-glance" clarity to make these categories more accessible to them. • In contrast, Environmental Health and Toxicology did seem more suited to both audiences. This discussion, however, yielded a finding that it was not obvious that there were two distinct sites. “Toxnet is only for free form search; it does not list topics or content. Toxnet does not seem like it is for the average joe … When I look at the list of databases I do not know or recognize the acronyms. Some people just want to know specific info and don't have time to learn along the way." (Laura, Citizen) "The acronyms throw you out of the search immediately, if you don't know what they stand for. It's not clear within a glance which site one would want to use. Toxnet seems better suited to professionals in .. chemistry, e.g." (Nancy, Citizen) "It’s actually not clear that it is two sites. I was confused while going from one site to the other...asking...am I on the same site... did I go to another site etc." (Linda, Citizen) "it was confusing knowing what site I was on at different times. Maybe sometimes open a new window, so I can still go back to the main portal and try something else?" (Shalom, Citizen)

Overall Findings: Environmental Health and Toxicology • Three Searches • Also unclear to many respondents (mostly in the Professionals group) was the fact that there are THREE searches on the site (SIS, NLM, and TOXNET). “I guess it wasn't clear to me---just chose toxnet and then NLM separately." (SueAnn, Professional) "Not clear to me either, why can't the search go through all of them automatically?" (Christopher, Professional) "SIS search isn't as clear because it doesn't have a blank box in which to type and the other two do. The NLM search is up in the header and I've probably already scrolled it off the page to see more content. Toxnet is clear - it is the topic of the page, it is set off, it lists databases, it has a search box." (Heather, Professional) "I was not clear on which to use." (Jim, Citizen)

Overall Findings: Environmental Health and Toxicology • "Especially For" • The majority of respondents in both groups felt that the "Topics" and "Especially For" sections were very useful starting points. Also appealing were the category list under "Topics" and the highlighting of NEW categories. • Some suggested adding categories to this area, such as breakdowns in the "Public" section for Parents, Kids, and Students. Other suggestions included "Community Volunteers," "Risk Assessors," and "Regulators." “Yes, I started my searches there practically every time.” (Ron, Professional) “I found a lot of links in the pesticide topic.” (Lorraine, Citizen) “Especially for is excellent, topics is good. The rest is overkill for the general public seeking a quick answer to a simple question.” (Mark, Citizen) “special populations, interest groups? College students? Are they on there?” (Juliana, Citizen) “might add sections for risk assessors and/or regulators." (SueAnn, Professional)

Overall Findings: Environmental Health and Toxicology • "More to Explore" • Most respondents also felt that the "More to Explore" section was useful. Some (though few) felt that the list was "too long" yet comprehensive. One user suggested adding a "Hot Topics" section to the list, and another suggested organizing the categories by purpose. • Again, in this area, the acronyms became an issue for some in the Concerned Citizens group, despite the aid of the mouse rollovers, and some in this group found the list to be long. "Organize by what the *user* is trying to do, rather than where the information resource originally came from." (Shalom, Citizen) “How to organize 'more to explore' … might have to simply rethink the page real estate. all valuable items but list is long. More to explore is a mixed list - some are individual resources and some are links to links.” (Heather, Professional) “I skipped a lot of more to explore seemed too lengthy and many of the topics unrecognizable as relevant.” (Lorraine, Citizen) “Although the acronyms are found on the home page in a hard-to-find manner (the question mark), they should be spelled out on each and every page in case someone links to a page or is referred.” (Mark, Citizen)

Overall Findings: Environmental Health and Toxicology • "Reference Tools" • Reference Tools received mixed reactions. It was the least explored area of the home page, but those who explored found the "news" and "A to Z" list particular useful. • Some in the Professionals group had questions. “The resources and the news-everything really-If any of these topics come up in my classes I am going to link them to this website." (Juliana, Citizen) “I liked the A-Z section.” (Robert, Citizen) “The reference tools seemed adequate for me.” (Mwenda, Professional) "Would it be too much to ask for OEL documentation info?” (Lawrence, Professional) "can one access MSDS info from the reference tools?” (Christopher, Professional)

Overall Findings: Environmental Health and Toxicology • Online Tutorials and Videos • The "online tutorial" concept was well received among Professionals, and -- while many Concerned Citizens showed an interest in this idea -- some in the Citizens group were skeptical. "Tutorials could be useful -- maybe Flash examples of using the site to find particular pieces of information." (Shyam, Citizen) "Would be useful for new PhD students." (Michael, Professional) "Didn't watch the tutorial but applaud it being there. We get … questions about importing data into all of the bibliographic management softwares … Could endnote and procite and refworks be added?" (Heather, Professional) "no but am interested in this because we recently have an institutional ‘copy’ of reference manager available; previously had to teach self a pkg Endnote which was time consuming" (SueAnn, Professional) "Would they provide value. Who would use this? Would the user pay to download the video? Who would product it, a professor looking to enhance his career on this area of interest, the government, or a 3rd party?" (Mark, Citizen) "That would be alright-but videos use up a lot of memory, power, etc-especially if you have an older computer or dial-up." (Juliana, Citizen)

Overall Findings: Environmental Health and Toxicology • Online Tutorials and Videos (cont'd.) • Several respondents said they would be unlikely to watch a video, however. For some, there was an issue with the player required to view the video. "I would not watch a video, just play around and enter search terms to find my info fast." (Lorraine, Citizen) "perfect...I like the online tutorials, videos sometimes slow the system down." (Robert, Citizen) "I didn't try it because my employer hasn't purchased reference manager software ..." (Nancy, Professional)

Overall Findings: Environmental Health and Toxicology • Trust and Reliability • In the Citizens group only, the issue of trust emerged among a minority of respondents, based on the fact that Environmental Health and Toxicology is a ".gov" site. • This view, however, was balanced by the majority of respondents in this group who DID trust the data. "yes for the most part but then most of it was govt sites and I don't completely trust the govt." (Lorraine, Citizen) "I'm sure it's tracked..." (Wendy, Citizen) "I don't trust gov't sites at all...they have an agenda obviously." (Linda, Citizen) "I would hope it would be. It seems like a reliable source" (Robert, Citizen) "Yes. I see it as a trustable source of info. And reliable." (Shyam, Citizen)

Overall Findings: Environmental Health and Toxicology • Anticipated Usage • With extremely few exceptions, an overwhelming majority of respondents expect to visit the Environmental Health and Toxicology site frequently and, in some cases, have already referred others to it. "Absolutely-I plan to do an in-service to update my staff." (Jon, Professional) “Definitely. I have already referred my mother." (Linda, Citizen) "Yes I would. I would make it a point to talk about it in our community meeting." (Shyam, Citizen) "I do recommend to my students as an easy-to-access source." (Michael, Professional)

Overall Findings: Environmental Health and Toxicology • Receiving Updates about Environmental Health and Toxicology • Most respondents in both groups prefer to learn about NLM resources via e-mail. "email updates are great … Perhaps at international scientific/medical meetings. Great stand at Eurotox last year." (Michael, Professional) "Email works fine. Also had a good experience at the NLM booth at a national conference." (Ron, Professional) "Had great exposure at APHA -- wish there had been a formal exposure here at EPA rather than we all discovering bit by bit, but that was what it was like 15-20 yrs ago." (Marcus, Professional) "Monthly email updates." (Robert, Citizen) "Thru' environment magazines. Thru' web engines." (Shyam, Citizen)

Overall Findings: Environmental Health and Toxicology • Receiving Updates about Environmental Health and Toxicology (cont'd.) • Some of the organizations mentioned as appropriate places to get the word out about Environmental Health and Toxicology included the following: • AAPCC • AACT • ALA • CDC • City/county governments • Home Owners Association • Hospitals • MLA • USAIN • Society of Toxicology • Association of Government Toxicologists • Professional organization for agriculture Following is a series of recommendations that emerged from the results of these online focus groups.

Recommendations • A few recommendations that emerged from these focus groups included the following: • Use mouse rollovers as well as on-screen definitions to explain acronyms used on the site. • Clarify the presence of the three search mechanisms (i.e., NLM, SIS, TOXNET). • Break "Public" categories under "Especially For" into sub-categories. • Clarify the points at which users leave one site and go to another. Though design consistency was a clear strength of the NLM sites, many users were unclear as to "where" they were. • In the same vein, providing site maps and other navigational tools (e.g., "bread crumbs") may significantly enhance the user experience. • In communicating the word about Environmental Health and Toxicology, canvas a broad range of both professional and non-professional venues. Most prefer receiving e-mail updates on new additions to the site. Following are respondent profiles of the participants in these online focus groups.

Background • Study Objectives • Methodology: Online Focus Group • Summary Conclusions • Overall Findings: Environmental Health & Toxicology • Recommendations • Appendix: Respondent Profile

Appendix: Respondent Profiles

Appendix: Respondent Profiles - Group 1 (Professionals)

Appendix: Respondent Profiles - Group 2 (Concerned Citizens)

Contact: Mary Beth Solomon For inquiries and capabilities on Qualitative Analytics 201.434.0404 917.601.1273

Qualitative Research Study on Environmental Health and Toxicology