1 / 22

Results of an Organizational Self-Assessment of Model Development and Application Practices

Results of an Organizational Self-Assessment of Model Development and Application Practices. Lora S. Johnson, Director of Quality Assurance. Leader Halûk Özkaynak. Workgroup Members. Subgroup B Alice Gilliland (Co-lead) Halûk Özkaynak (Co-lead) David Holland Larry Wymer

stone-livingston
Download Presentation

Results of an Organizational Self-Assessment of Model Development and Application Practices

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Results of an Organizational Self-Assessment of Model Development and Application Practices Lora S. Johnson, Director of Quality Assurance

  2. Leader Halûk Özkaynak Workgroup Members Subgroup B • Alice Gilliland (Co-lead) • HalûkÖzkaynak (Co-lead) • David Holland • Larry Wymer • Rogelio Tornero-Velez • Said Hilal Subgroup A • Justin Babendreier • Kenneth Schere • Lisa Melnyk (Lead) • Lora Johnson • Mitchell Kostich • Ram Vedantham • Valerie Zartarian

  3. National Exposure Research Laboratory • Mission • NERL conducts research and development that leads to improved methods, measurements and models to assess and predict exposures of humans and ecosystems to harmful pollutants and other conditions in air, water, soil, and food. • Six Divisions • Atmospheric Modeling Division • Ecosystems Research Division • Ecological Exposure Research Division • Human Exposure and Atmospheric Sciences Research Division • Microbiological and Chemical Exposure Assessment Research Division • Four geographic locations • Research Triangle Park, NC • Cincinnati, OH • Athens, GA • Las Vegas, NV • 450 FTEs

  4. Background • 2006 NERL began working on The Exposure Framework to facilitate • Strategic Planning • Organizational Development • Communication • 2007 NRC issued the report Models in Environmental Regulatory Decision Making • Modeling important to regulations • Recommended a stronger focus on life-cycle • Modeling practices must be responsive to the IQGs • 2007 NERL vetted and revised The Exposure Framework • 2007 Laboratory Director charted a workgroup to assess current practices related to modeling • 2009 Workgroup products completed • 20 p final report with 21 appendices

  5. Workgroup Activities • Inventory NERL models • Describe current practices relative to: • Model evaluation • Model refinement • QA • Peer review • Recommend best practices, taking into consideration: • CREM guidance • G5-M • NRC Report • Report on the potential of models to advance NERL exposure research, particularly with respect to: • Identification/prioritization of new research areas • Integration of methods, measurements, and models

  6. APPROACH • An initial meeting of workgroup members and NERL senior managers was held • Activities 1, 2, and 3 were assigned to Workgroup A • Activity 4 was assigned to Workgroup B • Entire organization was surveyed for existing models • What’s a model? • Some parts of the organization were unsure they had anything relevant to contribute • QA Managers summarized information in the QA tracking system • Both intramural and extramural modeling activities were included • Workgroup leader met with each of NERL’s six divisions to discuss both modeling and measurement research • Regular teleconferences were held • After a year, workgroup members met again with NERL senior managers

  7. Workgroup Products (1/2) Summary of Workgroup Findings and Recommendations • Appendix 1: Summary of Findings and Recommendations (ppt) • Appendix 2: Summary of Division Specific Research Integration Meetings (ppt) • Appendix 3: Recommended Future Activities for Refining Quality Assurance (QA) for Models (doc) NERL Model Survey • Appendix 4: NERL Model Survey Form (xls) • Appendix 5: NERL Model Survey Data Collated (xls) • Appendix 6: Summaries of EPA/ORD NERL Models based on the January 2008 NERL-Wide Survey of Models (doc) • Appendix 7: Current Evaluation, Peer-Review, and QA Practices for NERL Models (doc) Workgroup A Products • Appendix 8: NERL QA Managers Quality Assurance Report • Appendix 9: Summary of Guiding Principles and Best Practices for Quality Assurance in Modeling (doc) • Appendix 10: Summary of Life-Cycle model Evaluation Principles and Examples of NERL Models demonstrating their Implementation (doc) • Appendix 11: Life-Cycle Model Evaluation Example: CMAQ Model (ppt) • Appendix 12: Life-Cycle Model Evaluation Example: SHEDS Model (ppt) • Appendix 13: Life-Cycle Model Evaluation Example: 3MRA Model (ppt) • Appendix 14: Life-Cycle Model Evaluation Example: APTR Model (ppt)

  8. Workgroup Products (2/2) Workgroup B Products • Appendix 15: Summary of NERL Model Survey Data on Research Integration, Application and Linkage (pppt) • Appendix 16: Domains Covered by NERL Models over the Source-to-outcome Continuum (ppt) • Appendix 17: Data and Methodological Challenges in Linkage of Source-to-Dose Models to Health Information: PM and Ecological Goods and Services Examples (doc). • Appendix 18: Integrated Source-to-Outcome Figures Showing Model Linkages: PM Example (ppt) • Appendix 19: Integrated Source-to-Outcome Figures Showing Model Linkages: Ecological Goods and Services Example (ppt) Workgroup Presentations Outside of NERL • Appendix 20: CREM Meeting Presentation on NERL Modeling Workgroup Activities (ppt) • Appendix 21: SRA 2008 Conference Presentation on Model Evaluation at EPA: Responding to the NAS Report and Activities by the NERL Modeling Workgroup (ppt)

  9. Appendix 7: Current Evaluation, Peer-Review and QA Practices for NERL Models The current practices exercised by NERL Principal Investigators (PI) during model development and life-cycle were categorized into seven areas: • Model Documentation • Peer-Review • Model-to-Model Comparison • Model Validation by Comparison of Results to Data • Code Verification • Sensitivity and Uncertainty Analysis • Quality Assurance Planning

  10. QA Tracking System: Summary of Results The table below summarizes the information collected by the NERL QAMs.  The time period reviewed was FY05- FY07.  Most models were designated QA Category 4, the QA Category that requires the least extensive documentation and would not require independent evaluation.  However, the individual Division summaries indicate that independent evaluation is often used. Appendix 8 NERL QA Managers Quality Assurance Report

  11. Appendix 9: Summary of Guiding Principles and Best Practices for Quality Assurance in Modeling This summary provides an overview of various objectives, guidance, and best practices for modeling activities conducted within the NERL. The document draws upon the following four EPA guidance documents and a recent NRC report on models: • Guidance Document on Model Development, Evaluation and Application in Environmental Decision Making; Council for Regulatory Environmental Modeling (CREM), (EPA, 2009) • Integrated Information and Quality Management Plan (IIQMP); National Exposure Research Laboratory (NERL) and National Center for Computational Toxicology (NCCT), U.S. Environmental Protection Agency (EPA, 2005) • Guidance for Quality Assurance Project Plans for Modeling (EPA QA/G-5M); Office of Environmental Information (EPA, 2002a). • Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency; Office of Environmental Information (EPA, 2002b) – denoted also as EPA’s Information Quality Guidelines (IQGs). • Models in Environmental Regulatory Decision Making; National Research Council of the National Academies of Science (NRC, 2007)

  12. Appendix 3: Recommended Future Activities for Refining Quality Assurance (QA) for Models • Recommendation 1: Clarify requirements for documentation and peer-review for life cycle evaluation • Recommendation 2: Assign and periodically update QA categories of models: • Assign initial QA categories to models after evaluation of the recommended guidance for QA categories of modeling projects based on software and science considerations • review the QA category assignment for continued relevance on an annual basis

  13. Appendix 3: Recommended Future Activities for Refining Quality Assurance (QA) for Models (con’t) • Recommendation 3: Develop TQB criteria that recognizes the equivalency of model documentation with peer-reviewed journal articles • Recommendation 4: Create QA training for modelers • Recommendation 5: Make QA resources readily available to modelers • Recommendation 6: Support and assist with Agency efforts to modify G5-M, QA Project Plan Guidance for Models

  14. Appendix 15: Summary of NERL Model Survey Data on Research Integration, Application and LinkageSelected Results (1/4) Topic: Use of NERL Measurements, Methods and Databases in Model • If NERL database used, does model directly link with database or does the user manually update local databases with new information Results: • 41% of models employ measurements derived from NERL-sponsored data collection • Survey did not have sufficient resolution to draw conclusions on whether NERL-based methods were used in conjunction with NERL sponsored data collection • 70% of NERL models currently link with other databases • 7% of database updating is done automatically

  15. Model Survey Data - Selected Results (2/4) Topic: Current Users and Uses of Models Results: • Responses received for 35 of 37 known models • Three categories for current users: • used by any of the EPA offices (11/37 models) • used by a specific EPA office, e.g., NERL, Region (14/37 models) • other outside users, e.g., industry, academia, states (10/37 models) • Three categories for current uses: • Potential for further or linked use (18/37 models) • Extended for further uses (12/37 models) • No further development or extension (23/37 models)

  16. Model Survey Data - Selected Results 3/4 Topic: Future Users and Uses of Models Results: • Responses received for 35 of 37 known models • For ten of the models, neither users nor uses change in the future • Among the remaining 24, an expanded user group for 19 of the models and new uses for 14 were indicated • Nine models were felt to have potential for growth in terms of future users and use

  17. Model Survey Data – Selected Results 4/4 Topic: Model Linkages Model linkage assumes that one model provides output into another model Results: • 23/37 models (62%) are linked • 10/37 (27%) models are stand alone (i.e. not linked) • 4/37 models (11%) are missing or N/A responses • Of the 23 linked models, 10 (43%) are air, 8 (35%) are water, and 5 (22%) are multi-media

  18. Summary Findings and Recommendations • Unanimous support for models being important to NERL’s research program and providing a unique opportunity to better integrate the exposure research program • Most significant challenges: • Cross-divisional/cross disciplinary communications • Integrating methods, measurements, and models, in particular to address future research areas (e.g., climate, biofuels, nano) • Scientific, personnel and financial resource constraints • Strategic recommendations: • Implement a new planning process that is cross-cutting • Establish a standing NERL modeling group and advisory panel • Convene annual meetings of the NERL modeling group and senior management

More Related