Episode 3 - Ex-Com Final Report Tools & Validation

1. Episode 3 - Ex-ComFinal ReportTools & Validation Dr. Ralph Leemüller R&D Division - DFS

2. Overview Validation Methodology Innovative Validation Techniques and Tools Expert Groups Gaming Techniques Prototyping Sessions Modelling and Fast-time Simulations Combined Techniques Episode 3 - CAATS II Final Dissemination Event 2 Brussels, 13 & 14 Oct 2009

3. Concept Validation Methodology • Concept lifecycle (E-OCVM) • Use of E-OCVM • Episode 3 adressed Steps V1 – V3 • All validations in accordance to methodology • Validation strategy • Validation plans – Conduction of Exercise – Report 3

4. What is an Expert Group • Expert group working on a set of predetermined concepts in a structured manner • Gain from group expertise • specific professional profiles • using both the individual skills and the synergy of the group • More complete vision; • Larger number of factors; • More reliable • The mission of an Expert Group in a validation project is providing: • A reliable approach to real cases, even when assuming future situations; and • Experience based judgement analysis for supporting the feasibility of the developments to be undertaken.

5. Using Expert Groups • Pros • Concepts development, refinement and clarification; • Qualitative assessments; • Prediction of potential issues regarding future situations; • Support validation exercises design and development. • Analysis of consistency of validation results. • Cons • Difficulty to produce quantitative results; • Limited availability of the experts; • Biased vision; • Trend to keep things the way they are. • Outputs: • Operational details: refinement of DoDs and scenarios; • Refinement of the approach of the validation exercises; • Judgement of exercise results

6. Addressing the Concept Validation - Expert Groups Concept clarification and operationalfeasibility of processes ExpertGroup • Concept layered phases: • Expert Groups are suitable for the initial validation stage (concept definition and clarification) as well as to consider cross-functional issues, i.e. initial performance assessment and operational feasibility; • Pre-industrial development can be supported by experts. 6

7. What is Gaming • Human in the loop activity applicable to simulate decision making processes; • Experts are ‘around a table’ playing different roles as they were living real processes (AOCs, Pilots, AMC, Network Manager,...); • Every role has a particular scope, objectives and strategies during the process; • Scenario running in real-time (or even slow time); • Promoting non-routine thinking, thus enabling out-of-the-box thinking.

8. Using Gaming • Pros • Clarification of the concept; • Qualitative assessment of processes feasibility: • Detection of bottlenecks and shortcomings; • Clarification on roles and responsibilities: • Identification of the information flows among actors; • Exploration on supporting tools requirements. • Cons • Mostly qualitative assessment; quantitative results from SW-Games • Without an adequate guidance the Gaming session may turn into an Expert Group; • Results obtained during the game are dependant on the participants (expertise) and platform limitations. • Outputs: • Refinement of Operational details (Information flows, roles & responsibilities);

9. Supporting means to play • Three different gaming approaches: • Paper-based game; • Web-based game; • SW Platform-based: • DARTIS; ACCES; • CHILL; • PROMAS. 9

10. Addressing the Concept Validation – Gaming Concept clarification and operationalfeasibility of processes GamingExercise Paper-based Platform-based • Concept layered phases: • All phases can be covered by gaming (Planning and Execution Phases); • Suitable for clarification on components of the concept better than for global concept. 10

11. What is prototyping session Human in-the-loop experiments to build the ATMenvironment piece by piece and to address detailed issues (e.g. airspace, tools, roles). The focus is on operability: • Real-time technique • Small-scale and/or part-task • Iterative in nature • Gradually building and refining • Examine the options, moving towards a clear goal

12. Using Prototyping Session • Pros • Flexibility • Quick answers • Suited to early concept assessment/maturation • Reduced cost: small scale and simplified environment, to focus on specific issues • Cons • Short-time scales are the major limitation • One week sessions • Time between sessions • Training time • Examinination of the concept with multiple conditions • Outputs: • Assessment of operability and acceptability • refinement/maturation of a defined set of concept elements • Initial trends on some KPAs

13. Addressing the Concept Validation – Prototyping Session Operational feasibility Proto-typing • Concept layered phases: • Execution phases is covered by prototyping; • Suitable for feasibility on components of the concept better than for global concept. 13

14. What is Modellingand Fast-time Simulation • Using models of ATM systems • Use a representation of elements of the ATM concept • Applied when it is impractical to recreate the experimental conditions with human in the loop (large scope, long duration, wide area, …) • Predict measures related to physical processes or to assess feasibility of the processes.

15. Using Modelling and FTS • Pros • Large scope in time or geographic area • Short time of simulation run • Reduced cost - targeted use of expert • Reproducible • Quantitative results on KPA’s (i.e. Capacity, Efficiency) • Cons • Model agents may introduce simplifications • Analysis and interpretation requires an experienced expert • Level of maturity of concept influence the accuracy of output • Outputs: • Assessment of quantitative results (KPA) • refinement/maturation of a defined set of concept elements

16. Addressing the Concept Validation – Modelling and FTS Concept clarification and Feasibility of large scope, long duration Model & FTS Modelling Fast-Time • Concept layered phases: • All phases can be covered by gaming (Planning and Execution Phases). • Macro model to extrapolate results on ECAC-wide applicability • refinement/maturation of a defined set of concept elements • Suitable for concept assessment on wide area and large coverage. 16

17. Combining Techniques