1 / 20

Brooks, Legault, Nitschke, O’Brien, Sosebee, Rago, and Seaver

Nothing gives rest but the sincere search for truth. Blaise Pascal (French philosopher). WP 2.4 Evaluation of NMFS Toolbox Assessment Models on Simulated Groundfish Data Sets Comparative Simulation Tests Overview. Brooks, Legault, Nitschke, O’Brien, Sosebee, Rago, and Seaver. What did we do?.

arawn
Download Presentation

Brooks, Legault, Nitschke, O’Brien, Sosebee, Rago, and Seaver

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Nothing gives rest but the sincere search for truth. Blaise Pascal (French philosopher) WP 2.4Evaluation of NMFS Toolbox Assessment Models on Simulated Groundfish Data SetsComparative Simulation Tests Overview Brooks, Legault, Nitschke, O’Brien, Sosebee, Rago, and Seaver

  2. What did we do? • Evaluated 5 NFT stock assessment models for three stocks under 4 scenarios meant to examine potential difficulties in real assessments • AIM, ASPIC, SCALE, VPA, ASAP • GB yt (retro), GB cod (domes), white hake (ageing) • PopSim used to generate true conditions and create 100 datasets with the same random errors for all 5 models • Evaluated Accuracy and Precision of the 100 point estimates from the models • Did not examine precision of each of the 100 runs 60 scenarios 6000 assessments

  3. Why? • Test hypothesis that all models are impacted similarly when presented with the same underlying problem • A priori know that some models will not perform well under the test conditions because limiting data to VPA years • NOT trying to declare one model “winner” • NOT trying to declare any model “bad”

  4. PopSim Primer Age and Length Based Population Simulator User defines • Dimensions (Years, Ages, Plus Group Age, Lengths) • Initial NAA • Recruitment time series (or SRR) • Annual Fmult and selectivity • Biological Characteristics • M, von B, L-W • Fishery Sampling • Surveys • Sets Template for Stock Assessment Model

  5. Surveys vs Indices • Surveys • Are a property of the true population • Catchability defined for all ages and years • Uncertainty added to true values at age and length • Indices • Are a property of the model • Sum values from surveys • Can be either number or biomass based • Can be limited age range or entire age range • Can be changed between models without impacting underlying truth

  6. Growth • Initial NAA distributed according to stdev1 • Growth transfer matrices created for each age based on expected von B growth for age and stdev2 • Fish not allowed to decrease in size • Allows fishing to change distribution of length at age

  7. Market Sampling • Markets declared by user • Sampling conducted per 100 mt of landings in each market each year

  8. Input Output

  9. PopSim Limitations • PopSim is not reality • Annual Time Steps • Does not contain spatial components • Does not allow gender differences • Does not allow density dependent effects • No integrated management • Developing MSE wrapper to use PopSim, VPA, AgePro, and Control Rules

  10. This Exercise • Used utility to convert VPA run to PopSim • Gets Nyears, plus group age from VPA • Sets initial NAA and R from VPA • Sets annual Fmult from VPA • Estimates one logistic selectivity from VPA • Length and biology stuff from user • Market stuff from user • Surveys and Indices defs from user • Tuned markets, sampling, and surveys to represent actual assessments by lead

  11. Farmed Out Assessments • AIM – Rago • ASPIC – Brooks • SCALE – Nitschke • VPA – Legault, O’Brien, Sosebee • ASAP – Legault • Used base case to get template settings reasonable • Applied this base case to each of the test cases • Some models did additional runs with modified templates to “fix” the problem

  12. Results • PopSim compares the distribution of 100 assessments with the known true values • Exactly what is compared depends on model • E.g. VPA NAA & FAA, ASPIC B & F • Many, many runs and scenarios • PopSim creates tables and graphs • R program to gather results and automatically create plots

  13. Started by looking at direct results Black Line True Circles and Grey Line Median Red dashed Lines 5 and 95%iles

  14. Decided Bias and CV Better

  15. General Conclusions • Given failure of all models tested (simple & complex), we suspect other models would also be vulnerable to “retrospective agents” • Use of age-specific indices is robust to uncertainty in survey selectivity • If ageing is uncertain, these simulations support using models w/o age or models which allow uncertainty in catch at age

  16. General Conclusions (cont.) • VPA and ASAP ‘failures’ were similar in pattern • Magnitude of bias was less for ASAP • Precision usually somewhat better for ASAP • Given these similarities, we suggest that ASAP may offer some advantages to VPA (esp. in terms of flexibility)

More Related