results of an organizational self assessment of model development and application practices l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Results of an Organizational Self-Assessment of Model Development and Application Practices PowerPoint Presentation
Download Presentation
Results of an Organizational Self-Assessment of Model Development and Application Practices

Loading in 2 Seconds...

play fullscreen
1 / 22

Results of an Organizational Self-Assessment of Model Development and Application Practices - PowerPoint PPT Presentation


  • 102 Views
  • Uploaded on

Results of an Organizational Self-Assessment of Model Development and Application Practices. Lora S. Johnson, Director of Quality Assurance. Leader Halûk Özkaynak. Workgroup Members. Subgroup B Alice Gilliland (Co-lead) Halûk Özkaynak (Co-lead) David Holland Larry Wymer

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Results of an Organizational Self-Assessment of Model Development and Application Practices' - josef


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
results of an organizational self assessment of model development and application practices

Results of an Organizational Self-Assessment of Model Development and Application Practices

Lora S. Johnson, Director of Quality Assurance

workgroup members

Leader

Halûk Özkaynak

Workgroup Members

Subgroup B

  • Alice Gilliland (Co-lead)
  • HalûkÖzkaynak (Co-lead)
  • David Holland
  • Larry Wymer
  • Rogelio Tornero-Velez
  • Said Hilal

Subgroup A

  • Justin Babendreier
  • Kenneth Schere
  • Lisa Melnyk (Lead)
  • Lora Johnson
  • Mitchell Kostich
  • Ram Vedantham
  • Valerie Zartarian
national exposure research laboratory
National Exposure Research Laboratory
  • Mission
    • NERL conducts research and development that leads to improved methods, measurements and models to assess and predict exposures of humans and ecosystems to harmful pollutants and other conditions in air, water, soil, and food.
  • Six Divisions
    • Atmospheric Modeling Division
    • Ecosystems Research Division
    • Ecological Exposure Research Division
    • Human Exposure and Atmospheric Sciences Research Division
    • Microbiological and Chemical Exposure Assessment Research Division
  • Four geographic locations
    • Research Triangle Park, NC
    • Cincinnati, OH
    • Athens, GA
    • Las Vegas, NV
  • 450 FTEs
background
Background
  • 2006 NERL began working on The Exposure Framework to facilitate
          • Strategic Planning
          • Organizational Development
          • Communication
  • 2007 NRC issued the report Models in Environmental Regulatory Decision Making
          • Modeling important to regulations
          • Recommended a stronger focus on life-cycle
          • Modeling practices must be responsive to the IQGs
  • 2007 NERL vetted and revised The Exposure Framework
  • 2007 Laboratory Director charted a workgroup to assess current practices related to modeling
  • 2009 Workgroup products completed
          • 20 p final report with 21 appendices
workgroup activities
Workgroup Activities
  • Inventory NERL models
  • Describe current practices relative to:
      • Model evaluation
      • Model refinement
      • QA
      • Peer review
  • Recommend best practices, taking into consideration:
      • CREM guidance
      • G5-M
      • NRC Report
  • Report on the potential of models to advance NERL exposure research, particularly with respect to:
      • Identification/prioritization of new research areas
      • Integration of methods, measurements, and models
approach
APPROACH
  • An initial meeting of workgroup members and NERL senior managers was held
  • Activities 1, 2, and 3 were assigned to Workgroup A
  • Activity 4 was assigned to Workgroup B
  • Entire organization was surveyed for existing models
      • What’s a model?
      • Some parts of the organization were unsure they had anything relevant to contribute
  • QA Managers summarized information in the QA tracking system
      • Both intramural and extramural modeling activities were included
  • Workgroup leader met with each of NERL’s six divisions to discuss both modeling and measurement research
  • Regular teleconferences were held
  • After a year, workgroup members met again with NERL senior managers
workgroup products 1 2
Workgroup Products (1/2)

Summary of Workgroup Findings and Recommendations

  • Appendix 1: Summary of Findings and Recommendations (ppt)
  • Appendix 2: Summary of Division Specific Research Integration Meetings (ppt)
  • Appendix 3: Recommended Future Activities for Refining Quality Assurance (QA) for Models (doc)

NERL Model Survey

  • Appendix 4: NERL Model Survey Form (xls)
  • Appendix 5: NERL Model Survey Data Collated (xls)
  • Appendix 6: Summaries of EPA/ORD NERL Models based on the January 2008 NERL-Wide Survey of Models (doc)
  • Appendix 7: Current Evaluation, Peer-Review, and QA Practices for NERL Models (doc)

Workgroup A Products

  • Appendix 8: NERL QA Managers Quality Assurance Report
  • Appendix 9: Summary of Guiding Principles and Best Practices for Quality Assurance in Modeling (doc)
  • Appendix 10: Summary of Life-Cycle model Evaluation Principles and Examples of NERL Models demonstrating their Implementation (doc)
  • Appendix 11: Life-Cycle Model Evaluation Example: CMAQ Model (ppt)
  • Appendix 12: Life-Cycle Model Evaluation Example: SHEDS Model (ppt)
  • Appendix 13: Life-Cycle Model Evaluation Example: 3MRA Model (ppt)
  • Appendix 14: Life-Cycle Model Evaluation Example: APTR Model (ppt)
workgroup products 2 2
Workgroup Products (2/2)

Workgroup B Products

  • Appendix 15: Summary of NERL Model Survey Data on Research Integration, Application and Linkage (pppt)
  • Appendix 16: Domains Covered by NERL Models over the Source-to-outcome Continuum (ppt)
  • Appendix 17: Data and Methodological Challenges in Linkage of Source-to-Dose Models to Health Information: PM and Ecological Goods and Services Examples (doc).
  • Appendix 18: Integrated Source-to-Outcome Figures Showing Model Linkages: PM Example (ppt)
  • Appendix 19: Integrated Source-to-Outcome Figures Showing Model Linkages: Ecological Goods and Services Example (ppt)

Workgroup Presentations Outside of NERL

  • Appendix 20: CREM Meeting Presentation on NERL Modeling Workgroup Activities (ppt)
  • Appendix 21: SRA 2008 Conference Presentation on Model Evaluation at EPA: Responding to the NAS Report and Activities by the NERL Modeling Workgroup (ppt)
appendix 7 current evaluation peer review and qa practices for nerl models
Appendix 7: Current Evaluation, Peer-Review and QA Practices for NERL Models

The current practices exercised by NERL Principal Investigators (PI) during model development and life-cycle were categorized into seven areas:

  • Model Documentation
  • Peer-Review
  • Model-to-Model Comparison
  • Model Validation by Comparison of Results to Data
  • Code Verification
  • Sensitivity and Uncertainty Analysis
  • Quality Assurance Planning
appendix 8 nerl qa managers quality assurance report

QA Tracking System: Summary of Results

The table below summarizes the information collected by the NERL QAMs.  The time period reviewed was FY05- FY07.  Most models were designated QA Category 4, the QA Category that requires the least extensive documentation and would not require independent evaluation.  However, the individual Division summaries indicate that independent evaluation is often used.

Appendix 8 NERL QA Managers Quality Assurance Report
appendix 9 summary of guiding principles and best practices for quality assurance in modeling
Appendix 9: Summary of Guiding Principles and Best Practices for Quality Assurance in Modeling

This summary provides an overview of various objectives, guidance, and best practices for modeling activities conducted within the NERL. The document draws upon the following four EPA guidance documents and a recent NRC report on models:

  • Guidance Document on Model Development, Evaluation and Application in Environmental Decision Making; Council for Regulatory Environmental Modeling (CREM), (EPA, 2009)
  • Integrated Information and Quality Management Plan (IIQMP); National Exposure Research Laboratory (NERL) and National Center for Computational Toxicology (NCCT), U.S. Environmental Protection Agency (EPA, 2005)
  • Guidance for Quality Assurance Project Plans for Modeling (EPA QA/G-5M); Office of Environmental Information (EPA, 2002a).
  • Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency; Office of Environmental Information (EPA, 2002b) – denoted also as EPA’s Information Quality Guidelines (IQGs).
  • Models in Environmental Regulatory Decision Making; National Research Council of the National Academies of Science (NRC, 2007)
appendix 3 recommended future activities for refining quality assurance qa for models
Appendix 3: Recommended Future Activities for Refining Quality Assurance (QA) for Models
  • Recommendation 1:

Clarify requirements for documentation and peer-review for life cycle evaluation

  • Recommendation 2:

Assign and periodically update QA categories of models:

    • Assign initial QA categories to models after evaluation of the recommended guidance for QA categories of modeling projects based on software and science considerations
    • review the QA category assignment for continued relevance on an annual basis
appendix 3 recommended future activities for refining quality assurance qa for models con t
Appendix 3: Recommended Future Activities for Refining Quality Assurance (QA) for Models (con’t)
  • Recommendation 3:

Develop TQB criteria that recognizes the equivalency of model documentation with peer-reviewed journal articles

  • Recommendation 4:

Create QA training for modelers

  • Recommendation 5:

Make QA resources readily available to modelers

  • Recommendation 6:

Support and assist with Agency efforts to modify G5-M, QA Project Plan Guidance for Models

slide18
Appendix 15: Summary of NERL Model Survey Data on Research Integration, Application and LinkageSelected Results (1/4)

Topic: Use of NERL Measurements, Methods and Databases in Model

  • If NERL database used, does model directly link with database or does the user manually update local databases with new information

Results:

    • 41% of models employ measurements derived from NERL-sponsored data collection
    • Survey did not have sufficient resolution to draw conclusions on whether NERL-based methods were used in conjunction with NERL sponsored data collection
    • 70% of NERL models currently link with other databases
    • 7% of database updating is done automatically
model survey data selected results 2 4
Model Survey Data - Selected Results (2/4)

Topic: Current Users and Uses of Models

Results:

  • Responses received for 35 of 37 known models
  • Three categories for current users:
    • used by any of the EPA offices (11/37 models)
    • used by a specific EPA office, e.g., NERL, Region (14/37 models)
    • other outside users, e.g., industry, academia, states (10/37 models)
  • Three categories for current uses:
    • Potential for further or linked use (18/37 models)
    • Extended for further uses (12/37 models)
    • No further development or extension (23/37 models)
model survey data selected results 3 4
Model Survey Data - Selected Results 3/4

Topic: Future Users and Uses of Models

Results:

  • Responses received for 35 of 37 known models
  • For ten of the models, neither users nor uses change in the future
  • Among the remaining 24, an expanded user group for 19 of the models and new uses for 14 were indicated
  • Nine models were felt to have potential for growth in terms of future users and use
model survey data selected results 4 4
Model Survey Data – Selected Results 4/4

Topic: Model Linkages

Model linkage assumes that one model provides output into another model

Results:

  • 23/37 models (62%) are linked
  • 10/37 (27%) models are stand alone (i.e. not linked)
  • 4/37 models (11%) are missing or N/A responses
  • Of the 23 linked models, 10 (43%) are air, 8 (35%) are water, and 5 (22%) are multi-media
summary findings and recommendations
Summary Findings and Recommendations
  • Unanimous support for models being important to NERL’s research program and providing a unique opportunity to better integrate the exposure research program
  • Most significant challenges:
    • Cross-divisional/cross disciplinary communications
    • Integrating methods, measurements, and models, in particular to address future research areas (e.g., climate, biofuels, nano)
    • Scientific, personnel and financial resource constraints
  • Strategic recommendations:
    • Implement a new planning process that is cross-cutting
    • Establish a standing NERL modeling group and advisory panel
    • Convene annual meetings of the NERL modeling group and senior management