Design Approach Review NEXRAD Turbulence Detection Algorithm (NTDA)

Download Presentation

Design Approach Review NEXRAD Turbulence Detection Algorithm (NTDA)

Loading in 2 Seconds...

- 99 Views
- Uploaded on
- Presentation posted in: General

Design Approach Review NEXRAD Turbulence Detection Algorithm (NTDA)

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Design Approach Review

NEXRAD Turbulence Detection Algorithm (NTDA)

Gary Blackburn, Steven G. Carson, John K. Williams

NCAR Research Applications Laboratory

April 20, 2006

- CCR and goals
- Requirements addressed
- Operational concept
- Impact on external systems
- Dependencies on external changes
- Agency benefits
- NTDA algorithm diagram
- Development approach
- Real-time demonstration
- Data flow diagrams and AEL description
- Resource usage evaluation
- Test approach
- Integration challenges
- Proposed schedule
- Baseline documents to be updated
- Known issues

- CCR #NA06-09601“IMPLEMENTATION OF THE NEXRAD TURBULENCE DETECTION ALGORITHM (NTDA) IN THE NEXRAD OPEN RPG”
- Goals
- Supply turbulence data for the FAA AWRP’s Graphical Turbulence Guidance Nowcast (GTGN) product, scheduled for deployment in Spring 2010
- Display via WARP, CIWS, ITWS, and AWIPS (separate CCRs)

- Original WSR-88D turbulence detection requirement
- Commercial Aviation Safety Team Turbulence JSAT Report (January, 2001)
- FAA Aviation Weather Mission Needs Statement #339 Attachments 2 & 5 (June 2002)
- Commercial Aviation Safety Team JSIT report Enhancements 64, 65, 68, 72, 76 (July 2004)
- TAC website: “Turbulence analysis techniques” is NEXRAD Active Technical Need TN-17
- ~1300 FAA requirements for turbulence and convection
- However, there is currently no specific agency requirement for NTDA.

Polar Data Available to WARP, CIWS, ITWS, AWIPS

NEXRAD Turbulence Detection Algorithm

CONUS EDR Grids (NSSL mosaic at NCEP)

EDR Grids Available to Other Users

Convective Weather Nowcasts

Integrated Turbulence Diagnosis and Nowcast

In situ EDR

GTG Nowcast

(ADDS)

Satellite

Radar Reflectivity

NWP model (WRF)

GTG = Graphical Turbulence Guidance, ADDS = Aviation Digital Data Service (see http://weather.aero)

The current GTG clear-air turbulence forecast product with overlaid in situ turbulence reports from United Airlines aircraft

- Operational GTG: http://adds.aviationweather.gov
- Experimental GTG: http://weather.aero

- Data transmission issue
- How will data be transmitted/collected?
- Polar EDR and confidence data from each tilt (1 km range x 1 azimuth spacing) should be transmitted immediately to minimize latency
- AWIPS, “Level 2.5” product?

- Data collection, archival
- Display development considered a separate issue for users (separate CCRs for WARP, etc.)

- None known.

- FAA
- Direct benefit from products and displays
- Indirect benefit through commercial weather information providers, improved aviation situational awareness

- AFWA
- Similar to FAA benefits

- NWS
- EDR data may be useful for convection longevity diagnosis, or in NWP models

NTDA Algorithm Diagram(Fuzzy Logic approach)

- NTDA implemented in ANSI C
- Tested prototype in summer 2005 using LDM data from 16 NEXRADs
- 90-95% adherence to “Radar Product Generation System (RPG) C CODING STANDARDS, October 21, 2001”
- Raised CODE NTDA memory limit to 100 MB (current usage ~45 MB)
- No adaptation parameters for NTDA
- Currently uses CLAPACK, g2c, sunae.c (NASA)
- CODE prototype uses Build 7.0, public edition
- Will switch to Build 8.0, NWS edition when available

Dual-purpose implementation

Common Operations and

Development Environment /

Open Radar Products Generator

(CODE/ORPG)

Turbulence Remote Sensing

Realtime Demonstration System

(TRS/RDS)

Application:

CPC099/TSK001

ntda_alg

Application:

NCARTurbDetectAlg

NtdaAlgs API

NtdaAlgs_init

NtdaAlgs_FreeMemory

NtdaAlgs_CalcEdrFieldsInit

NtdaAlgs_EDR_SW_MEAN

NtdaAlgs_CalcEdrFieldsFree

Algorithm: ANSI C

NtdaAlgs.cc

Algorithm: ANSI C

NtdaAlgs.cc

Identical Code in Both

Environments

NTDA Real-Time Demonstration, Summer 2005

- Ability to use ORPG intermediate products
- Reduce the effort required to adapt the NTDA to accommodate ORDA upgrades
- Polar EDR data could be made broadly available to users (displays, NWP models, commercial vendors)
- Keep NTDA developers “in the loop” on baseline upgrades
- Involve ROC, OS&T, and other experts in design, implementation, and approval decisions

303 COMBASE_ELEV

ELEVATION_DATA

162 NTDA_EDR_IP

ELEVATION_DATA

160 NTDA_EDR

ELEVATION_DATA

NTDA_FP

99/02

ELEVATION

163 NTDA_CONF_IP

ELEVATION_DATA

299 RECCLDIGDOP

ELEVATION_DATA

161 NTDA_CONF

ELEVATION_DATA

NTDA_ALG

99/01

ELEVATION

NtdaAlgs – Data flow, Part 1

SW Multiplier

Table

One time

At startup

NtdaAlgs_init

params

struct

Application:

NCARTurbDetectAlg

or

ntda_alg

NtdaAlgs struct

Every

Elevation

range gates

artifact removal

field dimensions

spectrum width (SW)

radial velocity (VE)

azimuths & elevations

combined sweep

sun exclusion

NtdaAlgs_CalcEdrFieldsInit

array dimensions

reflectivity (DZ)

radial velocity (VE)

spectrum width (SW

signal-to-noise (SNR)

power-ratio (PR)

EDR

EDR confidence

SW variance

SW residuals variance

SW residuals sigma

SW confidence / SW variance

SW confidence / SW residuals

SW confidence / SW sigma

SW confidence / SW

SW confidence / SNR

SW confidence / Alt-DZ

SW confidence / PR

coverage fraction confidence

NtdaAlgs_EDR_SW_MEAN

NtdaAlgs – Data flow, Part 2

Every

Elevation

Application:

NCARTurbDetectAlg

or

ntda_alg

Netcdf file (TRS/RDS)

Or

Intermediate Products (ORPG)

allocate temp arrays

compute unsmoothed EDR^2

using range-dependent scaling

of SW

sun spike removal

smooth EDR^2

using 2D

confidence-weighted mean

reflectivity-derived

SW confidences

compute final EDR confidence;

encode all output

fields as short int

smooth SW confidence

if CombinedSweep

compute SW variance

compute variance of

SW residuals from

local linear fit

free temp arrays

Processing sequence

- Initialize quantities for this elevation
- OBTAIN Doppler bin size (meters)
- OBTAIN range to first Doppler bin (meters)
- OBTAIN unambiguous range (meters)
- COMPUTE Doppler data array size
- COMPUTE reflectivity array size
- COMPUTE array of ranges to reflectivity bins
- OBTAIN arrays of azimuth and elevation values
- for all radials in this elevation
- OBTAIN flag indicating whether this is a split-cut
- OBTAIN information needed to perform sunspike removal
- INITIALIZE reflectivity (DZ), signal-to-noise (SNR),
- power-ratio (PR) arrays to “missing data”
- OBTAIN DZ & SW arrays from COMBBASE_ELEV input buffer
- CALCULATE SNR from DZ
- CALCULATE PR from SNR
- COMPUTE arrays of ranges and azimuth spacing
- COMPUTE interpolate hi-res table of SW multipliers to
- construct array of SW multipliers for each range gate
- COMPUTE table of confidence as a function of range
- COMPUTE processing window size for EDR smoothing
- COMPUTE processing window size for SW variance calc
- COMPUTE processing window size for SW residuals calc
- COMPUTE processing window size for SW confidence smoothing
- COMPUTE remove artifacts from SW field
- INITIALIZE EDR and confidence Intermediate Product Buffers

AEL: Sun spike removal

2. Perform sunspike removal

COMPUTE azimuth and elevation of sun as seen from

radar location at current time

DO FOR ALL azimuths

IF beam azimuth & elevation are too close to sun

DO FOR ALL ranges

COMPUTE set DZ,SW,SNR,PR to “missing data”

END DO

END IF

END DO

AEL: Reflectivity-derived confidence

3. Calculate reflectivity-derived SW confidence

DO FOR ALL azimuths

COMPUTE sin and cos of beam elevation angle

DO FOR ALL ranges

COMPUTE spectrum width as floating-point value

IF reflectivity is not “missing data”

COMPUTE SNR as floating-point value

COMPUTE spectrum width confidence due to

SNR from fuzzy-logic interest map

COMPUTE spectrum width confidence due to

combination of altitude & DZ from

fuzzy-logic interest map

COMPUTE spectrum width confidence due to

PR from fuzzy-logic interest map

COMPUTE overall SW confidence due to all

DZ-derived sources by combining component

confidences using geometric average

ELSE // reflectivity is “missing data”

COMPUTE set all component confidences

to “missing data”

COMPUTE set overall reflectivity-derived SW

confidence to “missing data”

END IF

END DO // all ranges

END DO // all azimuths

4. Smooth reflectivity-derived SW confidence

IF this elevation is a “split-cut”

COMPUTE smooth DZ-derived SW confidence to account

for temporal incoherence

END IF

AEL: SW linear variance

5. Calculate variance of SW residuals after 2D linear fit

COMPUTE 2D linear fit of SW at each grid point using SVD

COMPUTE SW residual = difference between actual

and fit SW at each grid point

COMPUTE SW residual variance

COMPUTE Z-score of central residual

AEL: “Raw” EDR and confidence

6. Calculate EDR using range-dependent scaling of SW

DO FOR ALL azimuths

DO FOR ALL ranges

IF SW is not “missing data”

COMPUTE SW as floating point value

COMPUTE SW confidence due to SW residuals

variance using fuzzy-logic interest map

COMPUTE SW confidence due to SW residuals

sigma-score (standard deviations) using

fuzzy-logic interest map

COMPUTE SW confidence due to all SW-derived

confidences by combining component

confidences using geometric average

COMPUTE EDR confidence due to SW = product of

DZ-derived SW confidence times

SW-derived SW confidence

COMPUTE unsmoothed EDR from range-dependent

scaling of SW

COMPUTE unsmoothed EDR confidence =

EDR confidence due to SW

COMPUTE unsmoothed EDR squared = square of

Unsmoothed EDR

END IF // SW not “missing”

END DO // ranges

END DO // azimuths

AEL: Conf.-weighted mean EDR

7. Smooth EDR squared using confidence-weighted 2D mean

DO FOR ALL azimuths

DO FOR ALL ranges

COMPUTE size of averaging window in azimuth

and range based on range from radar

DO FOR ALL azimuths within averaging window

DO FOR ALL ranges within averaging window

IF EDR squared is not “missing”

COMPUTE sum of confidences

COMPUTE sum of EDR squared * confidence

END IF

END DO // ranges within window

END DO // azimuths within window

COMPUTE fraction of averaging window containing

non-missing data

COMPUTE confidence of averaged EDR squared as the

average of the confidences within window

COMPUTE confidence-weighted average of EDR squared

END DO // ranges

END DO // azimuths

AEL: Final product computation

8. Calculate final EDR and confidence and encode for output

DO FOR ALL azimuths

DO FOR ALL ranges

COMPUTE EDR = sqrt(EDR smooth squared)

COMPUTE CONF = EDR smooth squared confidence

COMPUTE confidence due to coverage fraction

using fuzzy-logic interest map

COMPUTE CONF = CONF * (coverage fraction conf)

IF SW not “missing”

COMPUTE encode EDR and CONF as short int

using scale factor and bias

ELSE // SW is “missing”

COMPUTE set EDR and CONF to “missing value”

END IF

COMPUTE store final EDR and CONF values

in output arrays

END DO // ranges

END DO // azimuths

- Test hardware
- dual 3.4 GHz Xeon processors
- 4 GB RAM, 2 MB FSB
- Red Hat Enterprise Linux 3

- Software: CODE Build 7.0, public edition
- Approach
- ran ORPG/CODE using 3 test cases of archived Level II data
- stripped-down NTDA (“best case”) and full version
- recorded total CPU time for NTDA
- evaluated size of BZIP2 compressed linear buffers (full 40-length buffer size = 29510272 bytes)

Case 1: KMKX 20050605 0700 UTC

NTDA CPU time per volume: 32s or 83s

NTDA compressed linear buffer sizes (length 40)

Turbulence EDR_IP: 497539 bytes

Confidence CONF_IP: 530829 bytes

VCP 121

Reflectivity (first tilt)

NTDA EDR

NTDA confidence

Case 2: KMKX 20050925 1900 UTC

NTDA CPU time per volume : 23s or 60s

NTDA compressed linear buffer sizes (length 40)

Turbulence EDR_IP: 360212 bytes

Confidence CONF_IP: 381658 bytes

VCP 21

Reflectivity (first tilt)

NTDA EDR

NTDA confidence

Case 3: KMKX 20050915 1600 UTC

NTDA CPU time per volume: 15s or 23s

NTDA compressed linear buffer sizes (length 40)

Turbulence EDR_IP: 170930 bytes

Confidence CONF_IP: 159110 bytes

VCP 32

Reflectivity (first tilt)

NTDA EDR

NTDA confidence

- Summer 2006 real-time demonstration
- continued statistical verification, pilot feedback
- case studies of poor performance events

- Run CODE version on LDM and archived data
- Perform extensive testing
- test CODE output to verify match with prototype
- stability/regression testing
- acceptance testing

- Make sure test suite includes volumes from each VCP

- NTDA developed/tested on Build 7.0/8.0, but intended for Build 10.0
- important to anticipate Build 10.0 changes that may impact the NTDA where possible

- Data format and transmission mode have not yet been determined

- SREC presentation: 2 May
- Summer 2006 demonstration, algorithm tuning: 1 June – 15 September
- Algorithm freeze: 15 September
- Code freeze: 1 October
- Intensive code testing/hardening period: 1 October – 30 November
- Second DAR: 15 October
- IRR: 5 December
- Code delivery: 5 January 2007

- System Specification (SS) – Yes
- Software Requirement Specification (SRS) – Yes
- Algorithm Enunciation Language (AEL) – Yes
- Product Specification – Yes
- Interface Control Document(s) (ICD(s)) – Yes
- Software Description Document (SDD) (VISIO diagrams) – Yes

- VCP 121 (MPDA) is not handled ideally
- adjust or eliminate use of PowerRatio confidence

- Beam indexing is not yet accommodated
- REC-based confidence is not yet integrated
- In CODE, NTDA is eventually killed, e.g.,
ERROR:NtdaApp:NtdaApp_Read_COMBBASE_ELEV: Mon Mar 13 16:19:26 2006

COMBBASE_ELEV pointer is NULL;

ERROR:NtdaApp:NtdaApp_Read_COMBBASE_ELEV: Mon Mar 13 16:19:26 2006

RPGC_get_inbuf status = 1;

ERROR:NtdaFpApp:NtdaFpApp_ReadInputs: Mon Mar 13 16:19:27 2006

NTDA_EDR_IP input buffer pointer is NULL;

ERROR:NtdaFpApp:NtdaFpApp_ReadInputs: Mon Mar 13 16:19:27 2006

RPGC_get_inbuf failed for NTDA_EDR_IP

RPGC_get_inbuf status = 1;

...