TPAWS AIRBORNE RADAR IN- SERVICE EVALUATION

1. TPAWS AIRBORNE RADAR IN- SERVICE EVALUATION PREPARED BY Dr. Roland L. Bowles AeroTech Research (U.S.A.), Inc.

2. HISTORY OF E-TURB. RADAR DEVELOPMENT ANALYSIS & SIMULATION NASA B757 FLT. EXP.’s IN-SERVICE EVAL. Program Start Dec./03 Hazard Characterization A/C Turb. Hazard Metric Detection Algorithm Dev. Hazard Prediction Algor. Dev. based On Radar Observables PDR Algorithm Perf. V&V Demonstrated Operational Feasibility & Technology Readiness WXR2100 Radar STC Installed Radar B737-800 (3708) R/C Sabreliner Flt. Test E-Turb. Algorithms FRD TPAWS R&T / Technology Transfer TSO Cert. Approval Eval. R/T Installed Commence ISE 8/23/04 2005 1998 2000 RLB

3. E-TURB RADAR – “STEP” Configuration “ SYSTEM TECHNOLOGY EVALUATION PROGRAM ” & PARTNERS Delta provided B737-800 NG aircraft ROCKWELL COLLINS Radar System Installed WXR-2100 multi - scan transceiver & antenna STC’s & TSO certified Collins advanced 2nd moment detection algorithm multi - lag auto-correlation ATR hazard prediction algorithm & B737- 800 hazard tables A/C weight to radar ATR Defined hazard criteria & display thresholds Fully Integrated cockpit display 2 level g - threshold “situation” display of advisory information E - TURB MODE CERTIFIED NON- INTERFERENCE WITH CURRENT WX-MODES / COCKPIT PROCEDURES RLB

4. E-TURB RADAR SYSTEM COMPONENTS RLB

5. E-TURB. System Implementation Concept FILTERED AIRCRAFT INPUTS WEIGHT & BALANCE TRUE AIRSPEED DETECTION ALGORITHM ALTITUDE ^  HAZARD PREDICTION ALGORITHM 2nd MOMENT PRODUCT ( i,j )  “g” units TO THRESHOLD LOGIC & DISPLAYS SPATIAL / TEMPORAL FILTERING “ g” - LOAD CALCULATION B737-800 RAW DOPPLER TURBULENCE MOMENTS AIRCRAFT HAZARD RLB

6. E-Turb.SYSTEM OPERATIONAL CONCEPT SUPPORTS TACTICAL AVOIDANCE OF CONVECTIVE TURBULENCE HAZARDS WHEN FEASIBLE • GENERAL • REQUIREMENTS • LOCATE • DETECT & QUANTIFY • PREDICT A/C SPECIFIC HAZARD • HAZARD SITUATION DISPLAY OF ADVISORY INFORMATION (seat belt-on) (seat belt-on avoid if possible) SITUATION DISPLAY RLB

7. EVENT DATA-LOGGER & DISPLAY THRESHOLDS DATA-LOGGER TRIGGERS: DATA SOURCES } Magnitude of peak IRU accel. ≥ . 5 g’s 5 sec. Windowed rms IRU accel. ≥ .156 g’s Radar predicted loads ≥ . 093 g’s Manual activation at crew discretion ADIRU#1 @ 50hz. (processed identical to TAPS) } RADAR DSP DISPLAY THRESHOLDS: } Level 1 (speckled magenta) ≥ .093 g’s Level 2 (solid magenta) ≥ .156 g’s RADAR DSP Jumpseat observations Crew questionaire response (ACARS free text) SUBJECTIVE EVAL. DATA : Note: Both radar & aircraft insitu data is recorded for each triggered event. Event data is recorded for all phases of flight from weight- off- wheels to weight- on- wheels. RLB

8. ISE EVENT CATEGORIZATION • Data for 408 events recorded for the period Aug. 23/04 – Feb. 11/05. • 63 events determined to be invalid* and data discarded. • 345 recorded events admissible for analysis. Radar Warn A/C Position Presence of Turb Yes Case No. In Warn Area Reflectivity IRU accel,Taps,Pirep 1 1 2 1 1 3 1 1 4 1 1 1 5 1 1 6 1 1 1 7 1 1 1 8 1 1 1 1 9 10 1 11 1 12 1 1 • * invalid criteria • touchdown “bumps” • radar in manual/ stand - by mode • low altitude with antenna pointing at ground in manual mode • inadvertent manual triggers • ...etc } N = 345 RLB

9. ISE EVENT DATA ANALYSIS METHODOLOGY N = 345 EVENTS • radar displays area of predicted turbulence • no radar display of predicted turbulence • aircraft flight path traverses affected area • cases 9-12 36 events • presence of turbulence verified by IRU accel. • indicative of CAT potential <11 % 122 events 184 events 35.1 % 54.2. % selected for correlation of radar prediction performance with actual experienced g - loads • radar displays area of predicted turbulence • aircraft does not traverse affected area RLB • presence of turbulence not verifiable by IRU data

10. DATA ANALYSIS ESTABLISH A PROBABLISTIC PREDICTION MODEL FOR THE SELECTED N=184 EVENT SAMPLING DISTRIBUTION Question: How well does x predict y and what confidence can we place on these estimates given all the operational variables present? ACTUAL EXPERIENCED INSITU LOADS y g’s RADAR PREDICTED LOADS x g’s RLB

11. Imagined distribution of insitu y values for 3 hypothetical radar x values conveying the idea of probability distributions of y about the line of means E(y/x) METHOD #1 Linear probabilistic model E(y/x) =(true regression) line of means . . . y observed insitu g’s ^ y = observed regression (y,x) = estimate of E(y/x) . . . x1 x2 x3 x radar predicted g’s Assumptions: We assume that for any given radar x value, the observed value of insitu y varies in a random manner and possesses a probability distribution with a mean value E(y/x). Inferences, significance & confidence intervals can be calculated assuming certain properties of the errors. RLB

12. 0.25 = b ( / ) E y x x raw data* 0.20 ˆ ˆ = = + b e ˆ y E ( y / x ) x pred y level1 display(.093 g's) 0.15 pred y level 2 display(.156 g's) 95% confidence intervals 0.10 0.05 0.00 0.00 0.05 0.10 0.15 0.20 0.25 CORRELATION OF RADAR PREDICTED & INSITU PEAK RMS g- LOADS N = 184 Turbulence Events ideal agreement Insitu Meas. g's ^ y = .819 x y r^2=.796 * raw data corrected for forward location of IRU accelerometer RLB Radar Predicted g's x

13. INFERENCES CONCERNING THE SLOPE ßOF THE REGRESSION LINE Question:Do the data present sufficient evidence to indicate that x (radar) contributes information for the prediction of y (insitu) over the region of observations? regression Test the null hypothesis H0: ß=0 against the alternative hypothesis Ha:ß≠0 . Test statistic: For ß´= 0 , which exceeds the critical value of t = 1.96 (183dof.) for 95% confidence. Observing that the test statistic strongly exceeds the critical value of t we reject the null hypothesis H0:ß=0, and conclude that there is convincing evidence to indicate that radar predictive results provide reliable information for the prediction of subsequent insitu g-loads. How much confidence can be placed in the estimate :For 95% confidence interval ß = .819 ± .0606 therefore .7584 ≤ ß ≤ .8796 . RLB

14. ESTIMATING E(Y/X),THE EXPECTED VALUE OF INSITU Y FOR A GIVEN VALUE OF RADAR X Estimating the mean value of y (insitu loads) for a given value of x (radar predicted loads) is an important practical problem, as well as finding a confidence interval for E(y/x). . Linear Probabilistic Model (“true” regression) P . Q . y (observed regression) (1 - a)% confidence interval for E(y/x) : R x For a given observation P(y,x) the true error is e(Q,P) Estimated residual is e(R,P) Difference between y and E(y/x) is = QR Key Assumption: errors are NID and µ = 0 ^ Example #1 Find a 95% confidence interval for the estimated mean insitu g-loads for a radar predicted x = .093 g’s which is the Level 1 display threshold for speckle magenta: .0701 ≤ E(y/x = .093) ≤ .0823 95% confidence; p= .05 Example #2 Same as example #1 except x = .156 g’s, the Level 2 display threshold for solid magenta: .1206 ≤ E(y/x = .156) ≤ .135 95% confidence; p= .05 RLB

15. Residuals -2 -1 0 1 2 Quantiles of Standard Normal Quantile-Quantile Plot of Residual Errors Vs. Standard Normal Diagnostic gives no reason to doubt the error residuals are normally distributed. 174 IMPLICATIONS 0.15 175 0.10 0.05 0.0 -0.05 Cumulative normal probability distribution 92 -0.10 rlb

16. SUMMARY STATISTICAL RESULTS • A t-test of null hypothesis ß=0 against alternative hypothesis ß≠0 provides strong & sufficient evidence to indicate that radar information provides reliable prediction of subsequent measured insitu g-loads with high confidence. • The linear radar prediction model explains about 80% of the variation in the observed insitu g-loads. • Analysis of residual error diagnostics give no reason to doubt the residuals are normally distributed (key theoretical assumption). • For the N=184 encounter data sample, the estimate of E(y/x) i.e. the mean value of insitu load for a given value of radar predicted load illustrates a general tendency of the radar predictions to over estimate between 11-24% for speckle magenta(x=.093g’s), and 13-23% for solid magenta(x=.156g’s) based on 95% confidence intervals. This performance is considered good given all the sources variability inherent in the system. RLB

17. 6 FOQA Incidents 3.0 18 ICAO Events 49 NASA EVENTS (02) 18 NASA EVENTS (00) 2.5 10 NTSB Accidents DAL 3708 (184 events) 2.0 D h UAL 747 Pacific Accident 1.5 two recent events 1.0 0.5 REGRESSION 0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 s D h Correlation of Peak Load With Peak RMS Load ( 5 sec. window) Based on Measurements for 291Turbulence Encounter Cases DATA SOURCES AIRCRAFT MIX / ALL FLIGHT PHASES 747 ' s 757 ' s y =- 1.86e-3 + 2.6614x R^2 = 0.945 767 ' s 737 ' s L1011 ' s DC10 ' s * Cases Chosen for Detail aal 757 * Modeling to Support FAA Peak l l Radar Certification Effort NASA Flt.* g's 191 - 06 NASA R232-10 xxx 737 * FOQA severe moderate extreme Peak g's ESTIMATED FROM RADAR OBSERVABLES FOR TPAWS CONCEPT RLB

18. DAL 3708 Encounter 08/24/2004

20. Excerpts of Jumpseat Observations on E-Turb Aircraft • “From 11,400 feet until about 9,000 feet, an area of predicted turbulence (concentrated speckles) within relatively low radar reflectivity correlated very well against what was experienced. Initially, the low reflectivity (green and nil returns) made both pilots skeptical about the existence of the turbulence, but after transiting the area, they agreed that the magenta had in fact told the truth.” • “Patch of speckled magenta encountered at FL290 within an area of relatively low reflectivity (green return), and all felt that the correlation between the turbulence predicted by the magenta and the turbulence actually experienced was very good. At first, the captain commented that perhaps this area had amounted to a false warning, since the magenta began to disappear behind the aircraft while the ride remained smooth. Before he was able to finish that assertion, the jolts that had been predicted occurred.”

21. Excerpts of Jumpseat Observations on E-Turb Aircraft(continued) • “Because of high radar reflectivity over ERLIN intersection, ATC vectored the flight 20 degrees left of course, bringing us directly into an area of low reflectivity (nil returns) but concentrated magenta speckles. Recognizing this, the first officer requested 20 degrees right of course as an alternative, which would have taken us very close to the heavier reflectivity at ERLIN but avoided turbulence (since there was no magenta depicted in that area). ATC denied the request, and there was good correlation between the speckles and what was experienced.” 4. “Together with TAPS, the enhanced turbulence radar effort marks one of the most exciting developments in the struggle to deliver better quality turbulence hazard information to flight crews and potentially other aviation user groups. While progress has so far been impressive, the program has little near-term relevance to the traveling public without a way for the enhanced turbulence software to be retrofitted on existing radar units, as well as for TAPS to continue its work. Tom Staigle Chief Technical Pilot Delta Air Lines

22. SUMMARY CONCLUSIONS • Basic WXR-2100 radar system performing without flaw. • E-Turb. functioning as per design & intended function. • Based on preliminary data, convincing evidence exists that crews are using E-Turb. to avoid indicated turbulence. • Preliminary data indicates good agreement between radar predicted loads and insitu truth when avoidance is not possible. • Documented flight crew feedback strongly supports the operational performance characteristics of the E-Turb. system design concept. RLB