NSW METOC Metrics – Next Steps

1. NSW METOC Metrics – Next Steps Presented by: Bruce W. Fordbruce@clearscienceinc.comhttp://www.clearscienceinc.com904-536-7180 This Brief is UNCLASSIFIED For official use only http://www.ClearScienceInc.com

2. Immediate Next Steps – Overview • Select 1-3 mission areas that are most likely candidates for the establishment of a data collection system • Diagram the mission flow process including each METOC and customer step including planning, (rehearsal),execution, and debrief. • Identify all discrete and non-discrete data elements provided by METOC personnel. Determine what elements are routinely verified. • Determine how the customer determines mission success. How is mission success recorded/reported? • Propose metrics to be collected (potentially multiple types) • Propose system for metrics collection/calculation/display For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

3. Select 1-3 mission types that are most likely candidates for the establishment of a data collection system • Recommend the most predicable mission type for initial data collection effort • Allows process challenges to be solved with “low hanging fruit” before attempting more difficult mission types. • Established planning/execution/debrief cycle • Established, stable METOC products • Recommend stable mission area • Predictable tactics • METOC products not in flux • METOC verification possible • Debriefing is routine • METOC inputs are part of planning process • METOC personnel are welcomed participants at debriefs For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

4. 2. Diagram the mission flow process including each METOC and customer personnel step including planning, execution, and debrief. • Recommend timeline format where possible • Annotate METOC inputs (routine/non-routine/formal/informal) to the cycle. For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

5. 3. Identify discrete and non-discrete data elements provided by METOC personnel. Determine what elements are routinely verified. • Use current METOC product examples • Discrete data element: • One forecast/one value • Ex: High Temp - 74 • Ex: Sig wave ht (bottom) – 5 ft • Ex: Sig wave ht (top) – 8 ft • Non-discrete data element: • One forecast/many potential values • Ex: Satellite image • Ex: Coastal current image • List discrete elements and identify verification sources • Anecdotal verification from customer • Scientific verification • Trained observers • Collection equipment For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

6. 4. Determine how the customer determines mission success. • Determine currently used customer measures of success • Binary (e.g., success/failure) • Non-binary (e.g., T1-T4) • Determine our ability to collect customer measures of success. Is automation possible? • Pre-mission message • Post mission message • Determine the extent of customer self-analysis • What mission information is archived by the customer? For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

7. 5. Propose metrics to be collected (potentially multiple types) Working definitions: • METOC performance metrics – Metrics of the quality of the METOC organization, including its operations, products, and services (e.g., accuracy of METOC forecasts of breaker height) • Customer performance metrics – Metrics of customer success (e.g., number of successful beach infiltrations) • Operational impacts metrics – Metrics of the impacts of METOC products on customer operations (correlation of breaker height forecast accuracy to beach infiltrations) • Proxy operational impacts metrics - Metrics of product performance that are acknowledged by METOC and customer personnel to be closely related to customer performance, so that the proxy operational impact metrics can be assumed to be good indicators of operational impacts (e.g., accuracy of breaker height forecasts which is assumed to be closely related to customer success) • Phenomena metrics – Metrics that relate product performance, customer performance, or operational impacts to specific environmental phenomena (e.g., breaker height forecast accuracy in the presence of high winds and seas; beach infiltrations in the presence of high winds and seas; correlation of breaker height forecast accuracy and beach infiltrations in the presence of high winds and seas) • Other examples of METOC metrics: • METOC Quality Metrics – Metrics of the overall quality of a product or service (e.g., forecast grade). These are a type of METOC performance metric. • METOC Capacity/Readiness Metrics – Metrics of organization’s operations (e.g., number of forecasters available, number of trained watchstanders). These are a type of METOC performance metric. • METOC Efficiency/ROI Metrics – Metrics of how well an organization uses its resources (e.g., number of forecasters required to publish forecast). These are a type of METOC performance metric. For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

8. 6. Propose system for metrics collection, calculation and display • Recommend SIPR web-based system • Minimize negative impact to METOC personnel • Strive for intuitive interface • Forecast collection • Verification input • Debrief input • Automate steps where possible • Message parsing • Forecast creator/collector • Verification collection • Strive to give METOC personnel a value-added tool • Streamline forecast process • Replace/enhance any current data collection efforts • Real time metrics calculation and display For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

9. Questions Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com