Session 11: Model Calibration, Validation, and Reasonableness Checks
Download
1 / 34

Source: NHI course on Travel Demand Forecasting ( 152054A) - PowerPoint PPT Presentation


  • 80 Views
  • Uploaded on

Session 11: Model Calibration, Validation, and Reasonableness Checks. Source: NHI course on Travel Demand Forecasting ( 152054A). Objectives:. Identify and interpret trends affecting travel demand Explain difference between calibration and validation Identify critical reasonableness checks

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Source: NHI course on Travel Demand Forecasting ( 152054A)' - jamal-tran


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Session 11: Model Calibration, Validation, and Reasonableness Checks

Source: NHI course on Travel Demand Forecasting (152054A)


Objectives
Objectives: Reasonableness Checks

  • Identify and interpret trends affecting travel demand

  • Explain difference between calibration and validation

  • Identify critical reasonableness checks

    • socioeconomic

    • travel survey

    • network

    • trip generation

    • mode split

    • trip assignment


Terminology
Terminology Reasonableness Checks

  • Model Calibration

  • Model Validation

    • Reasonableness checks

    • Sensitivity checks

  • Special generators

  • Screen lines (some modelers do not think this is important)

Is the model sensitive to policy options?


Key concepts
Key Concepts Reasonableness Checks

  • Not enough attention on model evaluation and reasonableness checks

  • Checks performed after each step

    • reduces error propagation

Errors can also “cancel”


Planner responsibilities
Planner responsibilities Reasonableness Checks

  • Actively involve all participants

    • Modelers

    • Planners

    • Decision makers

    • Public

  • Fairly present all alternatives

    • Timely

    • Unbiased

  • Identify (clearly) the decision making process

    • Who, when, and how

    • Allows input from all interested groups

  • You must rely on the TDM

    • Therefore, must be validated

    • Accurate and easy to understand (documented)


Trends affecting travel demand
Trends Affecting Travel Demand Reasonableness Checks

  • Planners should monitor the following trends:

    • Demographics

    • Composition of the labor force

    • Immigration and emigration

    • Regional economic development

    • Modal shares

    • Vehicle occupancy

    • Average trip length

    • Freight transport

  • Are trends consistent with assumptions made in the modeling process?

Must be aware of trends to ensure reasonable forecasts


Information requirements for validation and reasonableness
Information Requirements for Validation and Reasonableness Reasonableness Checks

  • Demographics and employment

  • Highway and transit networks

  • Model specification

  • Base year survey

  • Base year traffic counts


Sources of error
Sources of Error Reasonableness Checks

  • Coding

  • Sampling

  • Computation (if done by hand)

  • Specification

  • Data Transfer

  • Data aggregation

Improper structure of model, e.g., wrong variables


How do you judge a model recommend improvement
How do you judge a model/recommend improvement? Reasonableness Checks

Scrutinize these characteristics:

  • Data requirements

  • Hardware requirements

  • Logic of structure and conceptual appeal

  • Ease of calibration

  • Effectiveness of the model (accuracy, sensitivity)

  • Flexibility in application

  • Types of available outputs

  • Operational costs

  • Experience and successes to date

  • Public or private domain availability

  • Compatibility with other models and model types


Evaluation and Reasonableness Checks Overview Reasonableness Checks

Complete?

Level of Detail?

Reasonable?

Methodology?

Source?

Sensitive?

Documentation of calibration?

Valid for base year?

Current?

Reasonable?


Model Calibration and Validation Reasonableness Checks

Feedback Loop


Model validation
Model Validation Reasonableness Checks

  • Validation of new model

    • Model applied to complete model chain

    • Base year model compared to observed travel

    • Judgment as to model suitability, return to calibration if not

  • Validation of a previously calibrated model

    • Compare to a new base year, with new …

      • SE data

      • Special gen.

      • Network

      • Counts

“Transportation Conformity Guidelines” (Air Quality) require model validated < 10 years ago


Socioeconomic data check reasonableness
Socioeconomic Data: Check Reasonableness Reasonableness Checks

  • reviewSource for estimates and forecasts

  • Population and household size trends (graph 1950 to present and check trend)

  • Household income trends (graph as far back as this goes … 1990?)

  • Check dollar values used in forecast (use constant dollars)

  • If used, check trend of automotive availability (S curve?)

  • Check distribution of employment by type (basic, retail, service) over time

  • Plot and check trend of employees per household and per capita … rate of increase is decreasing

  • Check future household and employment changes by zone


Travel not sensitive to fuel price? Reasonableness Checks

http://www.eia.doe.gov/oiaf/aeo/pdf/trend_4.pdf


Travel survey data reasonableness checks
Travel Survey Data Reasonableness ChecksReasonableness Checks

  • Determine source of travel survey data

    • Types of survey conducted

    • Year of survey

  • If no survey (borrowed)

    • Check source of trip rates, lengths, TLFD

    • Is area similar

      • Geographic area?

      • pop/HH/empl. characteristics?

      • Urban density and trans system?

  • Compare to similar regions and to same

    region in earlier times:

    • Person trip rates by trip purpose

    • Mean trip lengths by trip purpose

      • HBW longest? HBO shortest?

    • TLFDs by trip purpose


Network data reasonableness checks
Network Data Reasonableness Checks Reasonableness Checks

  • Check Trees for 2-3 major attractions

  • Check coded facility types – how used (BPR?)?

  • Verify speed and capacity look-up table (what LOS used for capacity?)

  • Significant transportation projects – narrative included? Still viable?

  • Consistency with MTP

  • Plot (facility types, # lanes,

    speeds, area types) to detect

    coding errors


Trip generation reasonableness checks
Trip Generation Reasonableness Checks Reasonableness Checks

  • Examine trip production and attraction models

    • Form?

    • sensitivity?

  • Examine trip purposes used

  • External-through and external-local trips – how modeled?

  • Truck trips – how modeled?

  • Person trip or vehicle trip rates used?

  • P&A balance (0.9-1.1 ok)

  • Special generators (check, and be consistent in future model)


Trip generation calibration typical values
Trip Generation Calibration Reasonableness ChecksTypical Values

  • Person trips per household: 8.5 to 10.5

  • HBW person trips per household: 1.7 to 2.3

  • HBO person trips per household: 3.5 to 4.8

  • NHB person trips per household: 1.7 to 2.9

  • HBW trips: 18% to 27% of all trips

  • HBO trips: 47% to 54% of all trips

  • NHB trips: 22% to 31% of all trips


Trip generation calibration
Trip Generation Calibration Reasonableness Checks

TRIP PURPOSES Scaling Factor

HBW low income 0.795

HBW low-middle income 0.823

HBW middle income 0.861

HBW upper middle income 0.908

HBW high income 0.936

HB elementary school 0.733

HB high school 1.991

HB university 0.895

HB shopping 0.698

HB social-recreation 0.945

HB other 0.875

NHB work-related 0.858

NHB other 0.820

Truck 0.985

Internal-external 0.591

Note: each income class is a purpose!

Scale survey for participation (relative participation)

Colorado Springs 1996 Travel Demand Model Calibration


Trip generation calibration1
Trip Generation Calibration Reasonableness Checks

Reasonableness checks – compare to other cities, check future trends

  • Population 503,345

  • Households 201,116

  • Average Household Size 2.50

  • Basic employment 76,795 (33%)

  • Retail employment 50,465 (24%)

  • Service employment 101,697 (43%)

  • Military employment 42,800

  • Population per employee 1.81

  • Person trips per person 4.26

  • Person trips per household 10.65

  • HBW attractions per employee 1.44

  • HBW productions per household 1.74

  • HB shopping attractions per retail employee 5.99

Colorado Springs 1996 Travel Demand Model Calibration


Trip distribution reasonableness checks
Trip Distribution Reasonableness Checks Reasonableness Checks

Examine …

  • Mean trip length (increasing or decreasing?)

  • TLFDs

  • Treatment of friction factors (same?)

  • Treatment of terminal times (logic?)

  • Treatment of K factors

  • Comparison with JTW trip length

  • Comparison with JTW sector interchange volumes or percentages.


1 Reasonableness Checksst iteration

Calibrate friction factors


Calibrating a Gravity Model Reasonableness Checks

Adjusting Friction Factors


2 Reasonableness Checksnd iteration


Trip distribution calibration and validation
Trip Distribution Calibration and Validation Reasonableness Checks

  • Check modeled vs. household survey TLFD and mean trip lengths

  • Get HBW area-to-area flows from JTW

    HBW 1990 JTW TLFD and Area-to-Area Flows for Kansas City


Mode split reasonableness checks
Mode Split Reasonableness Checks Reasonableness Checks

  • Automobile occupancy factors by trip purpose used?

    • Basis?

    • Constant?

  • Mode split model?

    • Form?

    • Variables included in the utility functions?

      • Coefficients logical?

      • Value of time assumptions

      • Parking cost assumptions

  • How do mode shares change over time?

  • Mode share comparisons with other cities


Mode split calibration and validation
Mode Split Calibration and Validation Reasonableness Checks

  • Experienced planning consultant required …

    • Form of LOGIT model

    • Variables included in utility functions

    • Calibration of coefficients for utility function variables

    • Testing for IIA properties

    • Analysis of household survey data

    • Analysis of on-board transit survey data

  • Calibration tasks we can do:

    • Compare highway and transit trips

      • Total

      • By purpose

    • Compare Ridership by route

    • CBD cordon line survey (if bus service is downtown only)


Trip assignment reasonableness checks
Trip Assignment Reasonableness Checks Reasonableness Checks

  • All-or-nothing assignment

    • study effect of increasing capacity

    • Compare to Equilibrium assignment

  • Check volume delay equation (BPR parameters)

  • Compare

    • screen line volumes

    • Cut line volumes

  • Time-of-day assignments?

    • Source of factors

    • Peak spreading used for future?

    • If not, conversion factors source?

      (peak hour to 24-hour)

  • Local VMT (% assigned to

    intrazonals and centroid connectors

Equil

ibrium

All or

Nothing


Trip assignment calibration and validation
Trip Assignment Reasonableness ChecksCalibration and Validation

Assignment calibration performed

last

  • Overall VMT or VHT check

  • 40 to 60 miles per day per HH in large metro areas

  • 30 to 40 miles per day per HH in medium metro

  • +/- 10% OK on screen lines

  • Sign is important


  • Compute by … Reasonableness Checks

  • volume group

  • facility type

  • transit assignments

  • time of day


Other factors impacting forecasted travel demand use your noodle
Other Factors Impacting Forecasted Travel Demand (use your noodle)

  • Can be implied in travel surveys (but not explicit)

    • Telecommuting

    • Flexible work hours

    • HB business

  • How to account for …

    • Aging population

    • Internet shopping

    • Roadway congestion (will it affect generation in the future)

    • New modes


ad