MIW METOC Metrics: Ops Analysis

1. MIW METOC Metrics: Ops Analysis Tom Murphree and David Meyer Naval Postgraduate School (NPS) murphree@nps.edu Bruce Ford and Manuel Avila Clear Science, Inc. (CSI) bruce@clearscienceinc.com Paul Vodola, Matt McNamara, Luke Piepkorn, and Ed Weitzner Systems Planning and Analysis (SPA) pvodola@spa.com Presented at MIW METOC Metrics Symposium III CNMOC, Stennis Space Center, MS 1 February 2008

2. MIW METOC Metrics: Ops Analysis • Outline • Previous Ops Analysis Modeling • Weather Impact Assessment Tool (WIAT) • Linking METOC and Customer Performance • Role of Ops Analysis in the MIW Metrics Process MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

3. MIW METOC Metrics: Ops Analysis • WIAT – Overview • WIAT is • A pc-based, stochastic simulation tool used for assessing the impacts of METOC forecasts and METOC phenomena on strike operations • WIAT can • Simulate modern naval strike operations with a focus on fixed strike (STK) and kill box interdiction (KI) and close air-support (CAS) • Take as input both actual weather conditions and forecasts of conditions with varying lead times and accuracies • Emulate actual mission planning processes days and hours ahead of mission execution • Model customer decisions and outcomes based on weather predictions and other factors • Allows users to: • Simulate impacts of METOC forecasts and phenomena on spectrum of operational situations • Assess mission effectiveness employing a variety of METOC support situations • Determine the payoff/ROI for increased forecasting accuracy MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

4. 24% Observed and Forecasted Conditions Category % Weather Cancellations MIW METOC Metrics: Ops Analysis WIAT – Overview From NPS thesis research of LCDR Jake Hinz, USN, 2004 For official use only MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

5. MIW METOC Metrics: Ops Analysis • WIAT – Time-Error Forecast Analysis • Metrics collected through the online real-time collection system can be used as input to the ops analysis models to show the link between METOC performance and customer performance • Purpose of the following parametric analysis is to show that improved timing of forecasted features leads to improved operational effectiveness in Strike warfare • Perfect forecasting case is compared with timing errors of one and three days MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

6. 0 1 3 0 1 3 Time Shift (Days) MIW METOC Metrics: Ops Analysis METOC PERFORMANCE METRICS WIAT – Time-Error Forecast Analysis • Large amount of weather near the RYG thresholds decreases FAC & POD • Higher accuracy rate in forecasting Green Wx than Yellow or Red • Three-day errors not significantly worse than the one-day error MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

7. 0 1 3 0 1 2 3 Time Shift (Days) MIW METOC Metrics: Ops Analysis OPERATIONALMETRICS WIAT – Time-Error Forecast Analysis • Effectiveness per sortie is degraded due to inaccurate forecasting • Results in 11% decrease in targets killed per mission • Sorties with degraded BDA increased from 42 to 68 • Increase from 16% of all missions flown to 27% MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

8. Correlating METOC Support & Customer Performance 49% 42% Strike Mission Efficiency 39% 27% 23% Operational Performance 16% % Sorties with No BDA 40% 52% 100% Forecast Accuracy MIW METOC Metrics: Ops Analysis WIAT – Time-Error Forecast Analysis MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

9. Correlating METOC Support & Customer Performance 49% 42% Strike Mission Efficiency 39% • Where does the METOC community need to fall on the X-axis? • How can we make this determination? • NMETLS • NMAWC Metrics • Others? 27% 23% Operational Performance 16% % Sorties with No BDA 40% 52% 100% Forecast Accuracy MIW METOC Metrics: Ops Analysis WIAT – Time-Error Forecast Analysis MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

10. MIW METOC Metrics: Ops Analysis • Role in MIW Metrics • Develop and apply a set of operational analysis and modeling tools for: • Modeling MIW operations, especially METOC inputs to planning process and impacts on operational outcomes • Evaluating operational implications of METOC metrics based on real world data • Developing quantitative operational thresholds for the performance of METOC products • Identifying gaps and excesses in METOC support, and recommend mitigation strategies (DOTMLP-F) • Assessing alternative CONOPS for METOC support • Informing budgetary decisions at directorate and resource sponsor levels • Support initial operational use of tool set to provide assessment of METOC impact on customer performance MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

11. MIW METOC Metrics: Ops Analysis • Application • Tool allows for assessment of the impacts of METOC support products and services across a wide range of MIW scenarios • Determine sensitivities to increased Accuracy • Effect of enhanced Timeliness Scenario Multiple combinations of METOC products and services support MIW METOC Product MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

12. MIW METOC Metrics: Ops Analysis • Application • Use collected data to determine performance relative to the determined sensitivities • Answers the question “HOW WELL ARE WE DOING?” MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

13. MIW METOC Metrics: Ops Analysis • Application • For each Scenario – METOC Product pair, determine current performance in relation to the defined benchmarks • Identify gaps & excesses • Provide strategy recommendations (DOTMLP-F) MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

14. MIW METOC Metrics: Ops Analysis • Application Overview Periodic re-evaluation provides a feedback loop between collected data & performance benchmarks MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

15. MIW METOC Metrics: Ops Analysis QUESTIONS? MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

16. MIW METOC Metrics: Ops Analysis • Outline • Role of Ops Analysis in the MIW Metrics Process • Previous Ops Analysis Modeling • Weather Impact Assessment Tool (WIAT) • Linking METOC and Customer Performance MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

17. MIW METOC Metrics: Ops Analysis • Notional Tool • Notional MIW Weather Impact Assessment Tool • Simulates modern Mine Warfare operations • Inputs both actual environmental conditions and predictions with varying lead times and accuracies • Emulates actual mission planning processes days and hours ahead of mission execution • Models customer decisions and operational outcomes based on environmental predictions and other factors • Value • Simulates impacts of METOC predictions and phenomena on spectrum of MIW situations • Facilitates determination of phenomena which have significant impact on MIW operations • Evaluates MIW effectiveness employing a variety of METOC support situations MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

18. MIW METOC Metrics: Ops Analysis • Outline • Role of Ops Analysis in the MIW Metrics Process • Previous Ops Analysis Modeling • Weather Impact Assessment Tool (WIAT) • Linking METOC and Customer Performance MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

19. MIW METOC Metrics: Ops Analysis WIAT – Role in METOC Metrics Currently, WIAT has been used to experiment with various levels of METOC support to STW MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

20. Visibility MIW METOC Metrics: Ops Analysis WIAT – METOC Features • Forecasted and actual conditions over time • 0.5o spatial resolution • Temporal resolution: varying, down to 2 hrs • METOC features modeled • Cloud coverage (three layers) • Precipitation • Visibility • Wind speed • Includes inputs for other forecasted and observed conditions • Weather features impact all components of the simulations • Threat & friendly forces • Weapon probability of kill • Sensor probability of detection • Allows for the input of various forecast accuracies to capture operational effects of forecast uncertainty MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

21. 0 1 2 3 0 1 3 Time Shift (Days) Time Shift (Days) MIW METOC Metrics: Ops Analysis METOC IMPACTMETRICS WIAT – Time-Error Forecast Analysis • Mission change accuracy decreases by 40 – 60 percent • Over 30 fewer missions are saved when forecast timing errors are introduced MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

22. MIW METOC Metrics: Ops Analysis • Model Development Approach • Develop realistic vignettes and scenarios for MIW, including mission-level metrics of success • Identify the environmental factors that most significantly influence MIW platform performance and operational effectiveness • Describe and model the flow of METOC information from reachback, through in-theater support, to the MIW commander • Describe and model the MIW mission planning process, to include how and when METOC information is incorporated • Develop operational model(s) for MIW • Assess METOC support performance and impact on customer performance MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

23. ATO DRIVERS TACAIR Missions METOC Impacts on Operational Metrics Threat Forces Forecasted Weather • Changes Made During Planning Process Due to Weather Forecasts • Changes Made During Mission Due to Forecasts and Actual Weather • Mission Success • Opportunity Costs of Decisions Based on Forecasts and Actual Weather ATO Actual Weather Blue Ground Forces ISR Assets MIW METOC Metrics: Ops Analysis WIAT – Model Schematic MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

24. Resource (Sortie) Allocation Initial Plan Generation JPITL METOC Forecasting Products Strike Missions CAS Missions KI Missions Daily Target List ATO Mission (#AC, Weapons) ATO Mission (#AC, Weapons) ATO Mission (#AC, Weapons) ISR, FACs MIW METOC Metrics: Ops Analysis WIAT – Mission Planning Process Flight Plan Generation and Modification Timeline Flight Ops Execution Forecast Sortie Generation Planning Forecast • Secondary modificationsand sortie cancellations • Primary mission modifications • Initial flight plan generation Multiple Flight Plan Updates Over Time due to Forecasting & ISR MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

25. MIW METOC Metrics: Ops Analysis WIAT – Output Metrics • WIAT contains an automated output processing tool • WIAT provides an automated link between METOC support and operational performance WIAT Captures METOC Metrics… …and Operational Impact Metrics MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

26. MIW METOC Metrics: Ops Analysis • Application • Based on these determinations, METOC support benchmarks can be developed to establish goals and evaluate real world performance MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com

27. MIW METOC Metrics: Ops Analysis Linking METOC to Customer Performance WIAT – Time-Error Forecast Analysis • Forecast Accuracy with Three-day errors not significantly worse than the One-day error • Decrease in Forecast Accuracy leads to an increase in sorties that could not perform BDA due to Wx 0 1 3 Time Shift (Days) MIW METOC Metrics Ops Analysis, Feb 08, pvodola@spa.com