model certification
Download
Skip this Video
Download Presentation
Model Certification

Loading in 2 Seconds...

play fullscreen
1 / 19

Model Certification - PowerPoint PPT Presentation


  • 104 Views
  • Uploaded on

Model Certification. Jodi Staebell Operational Director, Ecosystem Restoration Planning Center of Expertise Rock Island, IL August 7, 2009. Model Certification. Process to review, improve, validate analytical tools and models

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Model Certification' - sonnagh


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
model certification

Model Certification

Jodi Staebell

Operational Director, Ecosystem Restoration Planning Center of Expertise

Rock Island, IL

August 7, 2009

slide2

Model Certification

  • Process to review, improve, validate analytical tools and models
  • Toolbox - Ensure high quality methods and tools available to enable informed decisions
  • Technical quality
    • Theory
    • Computational correctness
  • System Quality
  • Usability
slide3

Model Certification References

  • Information Quality Act (PL106-544)
  • OMB Information Quality Bulletin for Peer Review
  • Report of the Planning Models Improvement Task Force
  • Engineer Circular 1105-2-407
  • Updated EC under development
  • Protocols for Certification of Planning Models
  • HQ Memo Policy Guidance on Certification of Ecosystem Output Models, Aug 08
slide4

Model Certification Basics

  • Definition - any model or analytical tool used to
    • Define problems and opportunities
    • Formulate alternatives
    • Evaluate effects
    • Support decision-making
  • Planning models, not engineering models
  • Review is cost-shared
  • In-house or contracted
slide5

Model Basics

Stages of Model Development

  • Requirements stage
  • Development stage
  • External Testing
  • Implementation

Model categories

  • Corporate
  • Regional/local
  • Commercial off-the-shelf
  • Models developed by others
slide6

Model Documentation

  • Provided by model proponent
  • Documentation includes
    • Background
    • Theory, assumptions, analytical requirements, formulas
    • Software/hardware, testing/validation process,
    • Availability of input data, usefulness to support project analysis, tech support, training
  • Software/spreadsheets also provided
slide7

Model Review Team

  • Planner/Formulator
  • Functional Field Expert(s) (internal and/or external)
  • Software programmer/spreadsheet auditor
ecosystem output models flowchart to determine level of review
Ecosystem Output Models Flowchart to Determine Level of Review

Approved models include IWR Planning Suite and USFWS Published HSI Models

ATR Team reviews model application

Model certified/ approved for use?

Yes

No

IEPR required?

Detailed model review is required.

Yes

No

Single use or multiple use model?

Multiple Use

Detailed model review is required.

Single use of multiple use model?

Goal to approve application of model on specific project

Single Use

Single Use

ATR Team staffed and charged to conduct model review, supplement ATR team with outside experts, as appropriate

Multiple Use

Goal to recommend model for certification/ approval for use on future projects

Model Review$60-120k7-10 months

slide9

Model Certification Process

  • Develop Model Certification Plan
  • Assemble Review Team (In-house/Contract)
  • Charge to Reviewers
  • Kick-off Meeting
  • Conduct Review
  • Draft Model Review Report
  • Meeting to discuss findings
  • Final Model Review Report
  • Revise model, as needed (lather, rinse, repeat!)
  • PCX recommend that HQ certify model
model review process

District

MSC

Headquarters

PCX

Review Group

Model Review Process

District request PCX Model Review

  • Model Documentation
  • Published reports
  • Table 2 information
  • Model software or spreadsheet

PCX assign Model Review Manager

Model Review Plan approved by HQ

Model Review Plan

Scope/Cost/Schedule/Method

Model Review Contract Executed

Scope of Work

Independent Government Estimate

Contractor Cost proposal

Model Review Conducted

Initial Findings Meeting held

Draft Model Review Report

Draft Review Report submitted

Revised Model Documentation

Model and model documentation revised as needed

Final Model Review Report

Final Review Report submitted

Recommend model for certification/approval for use?

model review process continued

Recommend model for certification/approval for use?

No

Yes

HQ certify model/approve for use?

Yes

District

MSC

Headquarters

PCX

Review Group

Model Review Process (continued)

Model and model documentation revised as needed

or

PCX memo describing model limitations submitted to MSC

PCX recommend approval to HQ

PCX posts approved model and documentation in PCX Toolbox

slide12

Scheduling and Timing

Schedule

  • Start SOW to Notice to Proceed – 8-12 weeks
  • NTP to Final Model Review Report – 15-18 weeks
  • Revise model – depends on PDT
  • PCX recommendation to HQ – 4 weeks
  • HQ review and certify - ?

Timing

  • Requirements and/or development stage
  • Identify models at (prior to) Feasibility Scoping Meeting
  • Initiate model review prior to Alternative Formulations Briefing
slide13

Model Certification Cost

  • ATR – 3 reviewers extra time to review model
  • In-house - $30-50k
  • Contract - $80-140k
  • PCX labor ~$10-15k
  • Model proponent labor
    • Prepare model documentation
    • Assist in Model Cert contract
    • Revise model and documentation
slide14

Model Certification Experience

3 National models certified

  • IWR Planning Suite
  • Beach-FX
  • HEC-FDA

All published USFWS Habitat Suitability Index models approved for use

Many reviews of single use models

eco pcx model reviews
ECO-PCX Model Reviews
  • Island Community Index Model, NAB
  • 3 Everglades models
  • Upper Miss Fish Passage Effectiveness Index
  • Wetlands Value Assessment, MVN
  • Habitat Evaluation and Assessment Tools (HEAT) ERDC
  • Floodplain forest and wet coastal prairie community models, SWG
  • Des Plaines Community HSI and HGM, LRC
  • Arid Streams HGM, SPL
  • 4 Ecological Models, MVM
available models
Available Models
  • Ecosystem Restoration Gateway, Tools Tab
  • http://cw-environment.usace.army.mil/tools.cfm?CoP=Restore
slide18

How can the model proponent help?

  • Identify models early
  • Prepare model documentation
  • Test/validate model
  • Check software
  • Identify expertise needed
  • Be open to process
  • Pool funds for multiple-use models
ad