Hydrologic Modeling: Verification, Validation, Calibration, and Sensitivity Analysis Fritz R. Fiedler, P.E., Ph.D.
Definitions (review) • Verification: check if code solves equations correctly • Validation: check if model reasonably represents physical process • Calibration: adjust model parameters to match observations • Sensitivity Analysis: relative effect of parameter changes on output
Verification • Compare numerical results to analytical results
Level 1 Validation • Compare model results to simple experiments (can estimate parameters a priori)
Calibration • Adjust parameters to match observations
Level 2 Validation • Compare model results to observations for a different input data set post-calibration • Reserve some data (do not use in calibration) • After finding parameters that result in “best fit,” run model with reserved input and compare to output • Problems with this? • What happens in practice?
Sensitivity Analysis • Explore how parameter changes affect output • Sensitivity index:
Calibration Targets Can physically based model parameters be measured? Why or why not?
Goodness of Fit • Visual comparison between simulated and observed – look for trends in errors • A learned art • Use appropriate graph scales • Statistical performance measures • Consider mean daily discharge as calibration target • Q = observed • S = simulated
Means and Bias Common calibration strategy: fix bias first, revisit periodically, goal of no bias
Maximum Error: • Percent Average Absolute Error
Sum of Squares of Errors • Most common basis for statistical goodness of fit • e.g., least squares regression, seek to minimize
Root Mean Squared Error • Size of error usually related to size of events or values, thus RMSE typically smaller for dry periods, small watersheds (for example) • How would you modify RMSE to facilitate comparison?
Percent RMSE • Normalize RMSE by mean observed • Because the magnitude of RMSE varies with magnitude of values, by minimizing RMSE only, which part of hydrographs are primarily best fit in calibration? • How can this tendency be addressed?
Nash-Sutcliffe • Very popular method of evaluating calibration • Reading: McCuen, R. H., Evaluation of the Nash—Sutcliffe efficiency index, Journal of Hydrologic Engineering, 11(6), 597-602, 2006 (note: author uses different variables)
Line of Best Fit Analyze as in regression: hypothesis testing on A and B, residual analysis, correlation coefficient…
How to Use Statistical Measures • For a given time period, e.g., 1 year, and/or averages over multiple years • Look for seasonal trends
How to Use Statistical Measures • By flow interval (value interval) • Errors as f(Q) – aim for no systematic variation • How would you pick the intervals?
Exceedance Plots x x x x Q, S x x x x x x 0 percent days exceeded 100
Generalized Calibration Strategies • Set realistic parameter bounds before starting • Fix insensitive parameters first; focus on most sensitive • Eliminate most bias early in process, revisit • Use regionalized variables as appropriate • Combine manual and automatic techniques
Equifinality • Multiple combinations of parameters can lead to similar results • Issue with both multi-parameter lumped models (e.g., SAC-SMA) and spatially distributed models (e.g., CASC-2D) • Reading: Ebel, B. A. and K. Loague, Physics-based hydrologic-response simulation: Seeing through the fog of equifinality, Hydrological Processes, 20(13), 2887–2900, 2006