probabilistic nas platform
Download
Skip this Video
Download Presentation
Probabilistic NAS Platform

Loading in 2 Seconds...

play fullscreen
1 / 82

outline - PowerPoint PPT Presentation


  • 167 Views
  • Uploaded on

Probabilistic NAS Platform. George Hunter, Fred Wieland Ben Boisvert, Krishnakumar Ramamoorthy Sensis Corporation December 10, 2008. Outline. What is PNP? Team and development history Example uses of the model Software processes and testing Validation. Outline. What is PNP?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'outline' - Gabriel


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
probabilistic nas platform

Probabilistic NAS Platform

George Hunter, Fred WielandBen Boisvert, Krishnakumar Ramamoorthy

Sensis Corporation

December 10, 2008

outline
Outline
  • What is PNP?
  • Team and development history
  • Example uses of the model
  • Software processes and testing
  • Validation
outline3
Outline
  • What is PNP?
  • Team and development history
  • Example uses of the model
  • Software processes and testing
  • Validation
what is pnp
What is PNP?
  • An fast-time and flexible NAS-wide simulation tool
    • Real-time or fast-time modes
      • Half-hour runtime on a laptop, to simulate a day in the NAS
    • Physics-based: trajectories computed through integrating aerodynamic energy balance equations by varying the time-step size
    • System uncertainties (weather, security, operations …)
    • Plug-and-play architecture
      • Dynamic clients (TFM, DAC, AOC, …)
    • An ATC community resource
    • Formal software development processes in place
    • Adaptable to current system or NextGen future concepts
  • Uses
    • Environment in which to design, build and test decision support tools
      • TFM, DAC, AOC, …
      • Fast-time, real-time, shadow-mode
    • Potential NAS tool
      • Service provider, operator, collaborative uses
    • Benefits assessment tool
      • Fast-time tool to evaluate improved infrastructure, technology, procedures …
      • Evaluates historic and future traffic scenarios in weather
pnp architecture
PNP Architecture

Graphical User Interface

Plan View Display

NAS

Database

Reports

Flight Data

NAS Simulation

Probabilistic NAS Platform (PNP)

MATLAB®

Scripting

Interface

Weather Data

Performance Data

A fast-time physics-based (trajectory-based) NAS-wide modeling tool

pnp architecture6
PNP Architecture

Graphical User Interface

Plan View Display

NAS

Database

Reports

Flight Data

NAS Simulation

Probabilistic NAS Platform (PNP)

MATLAB®

Scripting

Interface

Weather Data

Performance Data

SimObjects

MATLAB®Client

Java Client

Client

As Middleware

Decision making

Prob-TFM

A fast-time physics-based (trajectory-based) NAS-wide modeling tool

External Client

(Any Language)

pnp client development
PNP Client Development
  • TFM client development
    • ProbTFM (Sensis internal development)
  • TFM client integration
    • C2 (algorithms from and used with permission of Bob Hoffman, Metron)
    • Constrained LP (algorithms from and used with permission of NASA, Joey Rios) in progress
  • DAC client integration
    • MxDAC (algorithms from and used with permission of Min Xue, NASA/UARC)
  • AOC client development
    • Gaming behaviors (collaboration with GMU/Lance Sherry) in progress
capabilities summary

Existing

Can Support

Capabilities Summary

  • Real-time
  • Fast-time
  • Airport weather impact models
  • Airspace weather impact models
  • Weather-integrated decision making
  • Probabilistic modeling / decision making
  • Traffic flow management
  • Dynamic airspace configuration
  • Surface traffic modeling
  • Terminal area modeling
  • Super density operations
  • Fuel burn modeling
  • Emissions modeling
  • Trajectory-based operations
  • Separation assurance
  • Plug-n-play
  • Fast run-time

outline9
Outline
  • What is PNP?
  • Team and development history
  • Example uses of the model
  • Software processes and testing
  • Validation

A fast-time physics-based (trajectory-based) NAS-wide modeling tool

team and development history

2003

2004

2005

2006

2007

2008

Huina

Gao

Krishnakumar

Ramamoorthy

Ben

Boisvert

Diego

Escala

Tae

Lee

Michelle

Lu

George

Hunter

WSI collaboration for real-time weather feed

NAS-wide,probabilistic

Wx modelingand routing

Clientarchitecture

Dynamic

clients

JPDO

FAA NASPAC

NASA NRAs

Internal

NWA

GMU

Team and Development History

People

Env’nment

Data

Funct’ality

Projects

Users

Project

System lead

Software lead

Software

System

Software

System

Java/real-time

Web 2.0

Matlab

outline11
Outline
  • What is PNP?
  • Team and development history
  • Example uses of the model
  • Software processes and testing
  • Validation

A fast-time physics-based (trajectory-based) NAS-wide modeling tool

example project uses
Example Project Uses
  • JPDO Modeling and Analysis
    • NextGen performance evaluation with weather
  • FAA NASPAC Weather Modeling
    • Convection impact modeling for NASPAC
  • NASA Gaming NRA
    • Evaluation of NextGen gaming with AOC clients
  • NASA MetaSimulation NRA
    • Investigation of TFM + DAC interactions
  • NASA SLDAST RFA
    • Evaluation of NextGen TFM concepts and models
  • NASA Market-Based TFM NRA
    • Evaluation of NextGen market-based TFM concepts
nextgen sensitivity studies

NextGen Performance Sensitivity Analysis

Benefit of Improved Wx Forecasts

Case 2: No distinction between clear and heavy weather forecast accuracy

Persistence forecast11/16/06

Case 1: Take advantage of improved forecast accuracy in clear weather

Benefit of Using Clear Weather Forecasts

NAS Performance Sensitivity

NextGen Sensitivity Studies

George Hunter, Fred Wieland " Sensitivity of the National Airspace System Performance to Weather Forecast Accuracy," Integrated Communications, Navigation and Surveillance Conference (ICNS), Herndon, VA, May, 2008

Kris Ramamoorthy, George Hunter, "Evaluation of National Airspace System Performance Improvement With Four Dimensional Trajectories," AIAA Digital Avionics Systems Conference (DASC), Dallas, TX, October, 2007

market based tfm studies

Delay

UAL233 Delay Cost

SCC

NAS Access Valuation Models

Market-Based TFM Studies

George Hunter, et. al., "Toward an Economic Model to Incentivize Voluntary Optimization of NAS Traffic Flow," AIAA ATIO Conference, Anchorage, AK, September, 2008.

dynamic airspace configuration

Nov 12, 2006, LAT=2, #Gen=40

ZFW FAA sectors

Dynamic Airspace Configuration

George Hunter, "Preliminary Assessment of Interactions Between Traffic Flow Management and Dynamic Airspace Configuration Capabilities," AIAA Digital Avionics Systems Conference (DASC), St. Paul, MN, October, 2008.

outline17
Outline
  • What is PNP?
  • Team and development history
  • Example uses of the model
  • Software processes and testing
  • Validation

A fast-time physics-based (trajectory-based) NAS-wide modeling tool

processes and testing cycle

Project Monitoring & Control

Quantitative Project Management

Development Tracking

Branch Configuration Management

Regression Testing

Unit and System Testing

Trunk Configuration Management

Processes and Testing Cycle

project monitoring and control
Project Monitoring and Control
  • JIRA is used to track issues
    • Project Manager and Lead Software Engineer assign task priorities, due dates, and personnel.
  • Weekly telecoms keep distributed team apprised of PNP and communications open
  • Project Manager maintains a master schedule in MS-Project
development tracking
Development Tracking
  • Software engineers use JIRA to track and status development efforts.
branch configuration management
Branch Configuration Management
  • Software Engineers are responsible for creating branches from the trunk to develop fixes/enhancements.
  • The Configuration Management of the software is accomplished with Subversion
    • Subversion is an open source version control system (http://subversion.tigris.org/)
unit and system testing
Unit and System Testing
  • Software Engineers are responsible for creating unit tests to verify the correctness of their code. The JIRA issue number is to be used throughout the code and unit tests for tracking purposes.
  • Software Engineers are responsible for running their own system/function tests to verify their software.
  • Once testing is validated, code is merged back on to the trunk.
trunk configuration management
Trunk Configuration Management
  • Once all validated JIRA issues are merged unto the trunk, regression testing is performed.
regression testing
Regression Testing
  • Regression testing
  • Aggregate results
    • Total delay
    • Total congestion
    • Traffic volume
    • #TFM initiatives
    • Runtime
  • Different scenarios
    • Truncated demand set
    • Full demand set
    • Weather
  • Automated
    • Weekly or as required
  • Archived
  • Graphical quick-look
quantitative project management
Quantitative Project Management
  • Regression testing validation is performed and the release letter is updated.
  • Release is tagged in Subversion.
  • JIRA issues are closed.
  • Documentation is updated to reflect changes in software.
outline26
Outline
  • What is PNP?
  • Team and development history
  • Example uses of the model
  • Software processes and testing
  • Validation

A fast-time physics-based (trajectory-based) NAS-wide modeling tool

system level engineering validation
System-Level Engineering Validation
  • ASPM / ETMS verification tests
    • Compare ASPM/ETMS data with simulation data
      • Calibrate concept to match aggregate field observations
    • Models
      • Trajectory data
      • Airport capacities (VMC / IMC)
      • Sector capacities in weather
    • Aggregate performance
      • Mean flight delay
      • Sector and airport overloadings
    • Detailed performance
      • Flight delay by airport and time of day
      • Overloading and delay patterns (Spatial and temporal)
        • Delays by airport and time of day
        • Sector and airport loading by time of day
        • Spatial loading patterns
    • Light and heavy weather days

system level software verification
System-Level Software Verification
  • Cross check sums
    • SFlights = SOperations at all airports
    • SFlight time = SMinutes from sector loads
    • SSector load by sector = SSector load by time
    • SAirport ops = SFlights using the airport in demand set
    • SDelays by flight = SDelays by time; and reroutes
  • Weather data checks
    • Compare PNP/Metar airport capacity with ASPM AAR/ADR
    • Compare PNP/Metar airport capacity with ASPM IFR periods
    • Ensure SEn route convection versus time of day is smooth
    • Ensure WxMAP ≤ MAP for all sector time bins
  • Graphical
    • Ensure reroutes overlaid on weather make sense
  • TFM Performance
    • Number of delays per flight, min and max flight delay
    • Maximum airport and sector overloading (ensure are reasonable)
system level engineering validation29
System-Level Engineering Validation
  • ASPM / ETMS verification tests
    • Compare ASPM/ETMS data with simulation data
      • Calibrate concept to match aggregate field observations
    • Models
      • Trajectory data
      • Airport capacities (VMC / IMC)
      • Sector capacities in weather
    • Aggregate performance
      • Mean flight delay
      • Sector and airport overloadings
    • Detailed performance
      • Flight delay by airport and time of day
      • Overloading and delay patterns (Spatial and temporal)
        • Delays by airport and time of day
        • Sector and airport loading by time of day
        • Spatial loading patterns
    • Light and heavy weather days

trajectory model validation

N: 316

Mean: 0.321 min

Std dev: 11.95 min

Detrended for Range

Mean: 0.80 min

Std dev: 6.51 min

R2: 0.012

Trajectory Model Validation
  • Compared to ETMS flight data (May 2008)

George Hunter, Ben Boisvert, Kris Ramamoorthy, "Advanced Traffic Flow Management Experiments for National Airspace Performance Improvement," 2007 Winter Simulation Conference, Washington, DC, December, 2007

probtfm performance
ProbTFM Performance
  • ASPM / ETMS verification tests
    • Compare ASPM/ETMS data with simulation data
      • Calibrate concept to match aggregate field observations
    • Models
      • Trajectory data
      • Airport capacities (VMC / IMC), actual and forecasted
      • Sector capacities in weather, actual and forecasted
    • Aggregate performance
      • Mean flight delay
      • Sector and airport loadings
    • Detailed performance
      • Flight delay by airport and time of day
      • Overloading and delay patterns (Spatial and temporal)
        • Delays by airport and time of day
        • Sector and airport loading by time of day
        • Spatial loading patterns
    • Light and heavy weather days

compare with field observations
Compare to ETMS/ASPM

Forecast accuracies, Decision making horizon, Delay distribution

January 7, 2007(Similar resultswith other days)

LAT = 60 minutes

(14.5,1657)

LAT = 30 mins

LAT = 0

Compare With Field Observations
verification of results
Verification of Results
  • ASPM / ETMS verification tests
    • Compare ASPM/ETMS data with simulation data
      • Calibrate concept to match aggregate field observations
    • Models
      • Trajectory data
      • Airport capacities (VMC / IMC), actual and forecasted
      • Sector capacities in weather, actual and forecasted
    • Aggregate performance
      • Mean flight delay
      • Sector and airport loadings
    • Detailed performance
      • Flight delay by airport and time of day
      • Overloading and delay patterns (Spatial and temporal)
        • Delays by airport and time of day
        • Sector and airport loading by time of day
        • Spatial loading patterns
    • Light and heavy weather days

system loading patterns

ETMS

ProbTFM

System Loading Patterns

ProbTFM predicted, 14:45 GMT

ETMS Actual, 14:45 GMT

ETMSUnderloading Overloading

ProbTFM loading

verification of results35
Verification of Results
  • ASPM / ETMS verification tests
    • Compare ASPM/ETMS data with simulation data
      • Calibrate concept to match aggregate field observations
    • Models
      • Trajectory data
      • Airport capacities (VMC / IMC), actual and forecasted
      • Sector capacities in weather, actual and forecasted
    • Aggregate performance
      • Mean flight delay
      • Sector and airport loadings
    • Detailed performance
      • Flight delay by airport and time of day
      • Overloading and delay patterns (Spatial and temporal)
        • Delays by airport and time of day
        • Sector and airport loading by time of day
        • Spatial loading patterns
    • Light and heavy weather days, control days

conclusion
Conclusion
  • The development of PNP has benefited from lessons learned over past two decades in NAS system wide modeling
    • Plug and play simulation architecture
    • Supports both analytical and HITL studies
    • Adaptable to simulate current system as well as NextGen future concepts
    • Fast-time, physics-based
    • Formal software development processes in place
    • Probabilistic decision making and extensive weather modeling explicitly incorporated in tool
publications
Publications
  • George Hunter, "Preliminary Assessment of Interactions Between Traffic Flow Management and Dynamic Airspace Configuration Capabilities," AIAA Digital Avionics Systems Conference (DASC), St. Paul, MN, October, 2008.
  • George Hunter, et. al., "Toward an Economic Model to Incentivize Voluntary Optimization of NAS Traffic Flow," AIAA ATIO Conference, Anchorage, AK, September, 2008.
  • George Hunter, Fred Wieland " Sensitivity of the National Airspace System Performance to Weather Forecast Accuracy," Integrated Communications, Navigation and Surveillance Conference (ICNS), Herndon, VA, May, 2008.
  • George Hunter, Kris Ramamoorthy, "Integration of terminal area probabilistic meteorological forecasts in NAS-wide traffic flow management decision making," 13th Conference on Aviation, Range and Aerospace Meteorology, New Orleans, LA, January, 2008.
  • Kris Ramamoorthy, George Hunter, "The Integration of Meteorological Data in Air Traffic Management: Requirements and Sensitivities," 46th AIAA Aerospace Sciences Meeting and Exhibit, Reno, NV, January, 2008.
  • George Hunter, Ben Boisvert, Kris Ramamoorthy, "Advanced Traffic Flow Management Experiments for National Airspace Performance Improvement," 2007 Winter Simulation Conference, Washington, DC, December, 2007.
  • Kris Ramamoorthy, George Hunter, "Evaluation of National Airspace System Performance Improvement With Four Dimensional Trajectories," AIAA Digital Avionics Systems Conference (DASC), Dallas, TX, October, 2007.
  • Kris Ramamoorthy, Ben Boisvert, George Hunter, "Sensitivity of Advanced Traffic Flow Management to Different Weather Scenarios," Integrated Communications, Navigation and Surveillance Conference (ICNS), Herndon, VA, May, 2007.
  • George Hunter, Ben Boisvert, Kris Ramamoorthy, "Use of automated aviation weather forecasts in future NAS," The 87th American Meteorological Society Annual Meeting, San Antonio, TX, January, 2007.
  • Kris Ramamoorthy, George Hunter, "Probabilistic Traffic Flow Management in the Presence of Inclement Weather and Other System Uncertainties," INFORMS Annual Meeting, Pittsburgh, PA, November, 2006.
  • Kris Ramamoorthy, Ben Boisvert, George Hunter, "A Real-Time Probabilistic TFM Evaluation Tool," AIAA Digital Avionics Systems Conference (DASC), Portland, OR, October, 2006.
  • George Hunter, Kris Ramamoorthy, Alexander Klein "Modeling and Performance of NAS in Inclement Weather," AIAA Aviation Technology, Integration and Operations (ATIO) Forum, Wichita, KS, September 2006.
  • Kris Ramamoorthy, George Hunter, "A Trajectory-Based Probabilistic TFM Evaluation Tool and Experiment," Integrated Communications, Navigation and Surveillance Conference (ICNS), Baltimore, MD, May, 2006.
  • Kris Ramamoorthy, George Hunter, "Avionics and National Airspace Architecture Strategies for Future Demand Scenarios in Inclement Weather," AIAA Digital Avionics Systems Conference (DASC), Crystal City, VA, October, 2005.
  • George Hunter, Kris Ramamoorthy, Joe Post, "Evaluation of the Future National Airspace System in Heavy Weather," AIAA Aviation Technology, Integration and Operations (ATIO) Forum, Arlington, VA, September 2005.
  • James D. Phillips, “An Accurate and Flexible Trajectory Analysis,” World Aviation Congress (SAE Paper 975599), Anaheim, CA, October 13-16, 1997.
pnp systems requirements
PNP Systems Requirements
  • System requirements
    • PNP is a Java application
    • Hardware
      • Memory: minimum 1GB, preferred 2GB
      • CPU: Pentium (4) 3.2 GHz or better
      • Video card: 128MB memory, preferred 256MB
    • Software
      • Java JDK 6 http://java.sun.com/javase/downloads/index.jsp
      • MySQL Server 5.0 http://dev.mysql.com
    • Third party licenses
      • Eurocontrol BADA usage license
weather days42
Weather Days
  • Weather days
    • Spectrum of weather days
      • Variation in weather type and intensity
      • Variation in season
    • Support real-world comparison
      • Support same sector data
      • Variation in traffic demand volume and structure
        • Different days of week, holidays
      • Control days
benefit evaluation
Benefit Evaluation

Case 2: No distinction between clear and heavy weather forecast accuracy

Case 1: Take advantage of improved forecast accuracy in clear weather

Persistence forecast, 11/16/06

congestion delay relationship
Congestion-Delay Relationship
  • Unconstrained sector congestion cost (SCC) for zero lookahead time (blue) and PNP-ProbTFM simulated delay (black) time histories for all en route NAS sectors and flights, respectively.

Delay

SCC

aggregate delay model
Aggregate Delay Model
  • Hypothesize a first-order lag transfer function

Simulated delay

Modeled delay

aggregate delay model56
Aggregate Delay Model
  • Hypothesize a second-order transfer function

Simulated delay

Modeled delay

explicit cost model
Explicit Cost Model
  • Evaluate cost of NAS access by removing the flight
  • Remove one flight
    • 11/16/06, UAL233, A320
    • Morning departure from Bradley International (KBDL) to Chicago O’Hare airport (KORD)
    • Relatively high cost flight
      • 90.02 SCC
remove ual233
Remove UAL233
  • Delay reduction by time bin in simulation run
    • Delay reduction of 8141 minutes
nas performance sensitivity studies

Nov 12, 2006

ETMS/ASPM

Non AgileDelay Distribution

Delay Distribution

Non AgileMinimum Delay

Minimum Delay

NAS Performance Sensitivity Studies
  • Performance sensitivity to:
      • Delay distribution policy (most important factor)
      • TFM system agility
      • System forecasts (least important factor)
nas sectorization
NAS Sectorization
  • Nov 12, 2006
mxdac afternoon sectorization
MxDAC Afternoon Sectorization
  • Nov 12, 2006, LAT=6, #Gen=20
mxdac midday sectorization
MxDAC Midday Sectorization

Coeff_peak_ac_var=0.0

Coeff_avg_ac_var=0.0

Coeff_crossings=0.0

Coeff_transition_time=0.0

Coeff_residual_capacity=1.0

  • Nov 12, 2006, LAT=2, #Gen=40
cost of distributing delay

Nov 12, 2006

$65/minute

Increased delay distribution

Cost of Distributing Delay
  • RMS delay can be reduced by spreading delay to more flights
    • But at the cost of increased total delay
project monitoring and control75
Project Monitoring and Control
  • JIRA is used to track issues
    • Project Manager and Lead Software Engineer assign task priorities, due dates, and personnel.
  • Weekly telecoms keep distributed team apprised of PNP and communications open
  • Project Manager maintains a master schedule in MS-Project
development tracking76
Development Tracking
  • Software engineers use JIRA to track and status development efforts.
branch configuration management77
Branch Configuration Management
  • Software Engineers are responsible for creating branches from the trunk to develop fixes/enhancements.
  • The Configuration Management of the software is accomplished with Subversion
    • Subversion is an open source version control system (http://subversion.tigris.org/)
unit and system testing78
Unit and System Testing
  • Software Engineers are responsible for creating unit tests to verify the correctness of their code. The JIRA issue number is to be used throughout the code and unit tests for tracking purposes.
  • Software Engineers are responsible for running their own system/function tests to verify their software.
  • Once testing is validated, code is merged back on to the trunk.
trunk configuration management79
Trunk Configuration Management
  • Once all validated JIRA issues are merged unto the trunk, regression testing is performed.
regression testing80
Regression Testing
  • Regression testing
  • Aggregate results
    • Total delay
    • Total congestion
    • Traffic volume
    • #TFM initiatives
    • Runtime
  • Different scenarios
    • Truncated demand set
    • Full demand set
    • Weather
  • Automated
    • Weekly or as required
  • Archived
  • Graphical quick-look
quantitative project management81
Quantitative Project Management
  • Regression testing validation is performed and the release letter is updated.
  • Release is tagged in Subversion.
  • JIRA issues are closed.
  • Documentation is updated to reflect changes in software.
risk management
Risk Management
  • Lessons learned analysis
    • A wrap up meeting is held to discuss all issues on a project in which proactive steps can be documented to avoid the same mistakes
ad