1 / 26

Procedures for FOD Detection System Performance Assessments

Procedures for FOD Detection System Performance Assessments. Radar-Based and Dual Sensor Systems. Beth Woodworth Edwin Herricks University of Illinois. Introduction. I will be talking today about two performance assessment efforts.

dyanne
Download Presentation

Procedures for FOD Detection System Performance Assessments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Procedures for FOD Detection System Performance Assessments Radar-Based and Dual Sensor Systems Beth Woodworth Edwin Herricks University of Illinois

  2. Introduction • I will be talking today about two performance assessment efforts. • The initial effort began in 2004 and was completed with assessment of an installed system on a runway at PVD in 2008. • The second began in 2007 and was completed in 2009.

  3. Sensors Evaluated • Qinetiq TarsierTM • Radar based system • Installed at PVD • Xsight FODetectTM • Hybrid system • Installed at BOS

  4. Testing of QinetiQ TarsierTM at PVD • 94Ghz radar • Detects FOD at ranges up to 1 km • 2 sensor units placed on towers scanned Runway 5/23 at PVD • Primary performance criterion was a standard -20dBm2 target at a range of 1km

  5. Testing of FODetectTM at BOS • 74GHz radar and video camera with image processing • Sensors replaced edge lights • Partial installation at BOS • Sensors covered 550 ft on Runway 15R • 10 surface detection units (SDUs) – 5 SDUs on each side of the runway • Each sensor covered approximately 75ft x 200ft • Primary performance criterion detection of 0.8 in (2 cm) object

  6. General Assessment Protocol • Testing scheduled over approximately 12-month period to capture varied weather conditions • PVD – June 2007- March 2008 • BOS – June 2008 - March 2009 • Targets selected to challenge each of the systems and included • Different sizes • Different shapes • Different materials • Different hazard potentials

  7. Detecting a Target • Radar • Reflectivity – energy returned to sensor • “Complex Scatterers” – most FOD has a variety of edges which have different likelihoods of being detected. • Electro-Optical • Background • Item surface conditions • Color, Flat/Glossy • Illumination – • Lighting and shadow

  8. Overall Goals of Testing • Calibration – Standard Target Testing • Uses items of known detection characteristics • Provides information on system reliability and robustness • Performance • Group of FOD items with different hazard potentials • Repeatedly placed at known locations in defined positions • Blind • Random placement and actual and simulated FOD items • This test most represents typical detection needs at an airport • Challenges system to detect a diverse and unpredictable set of items.

  9. General Methods • Safety – • Because assessments were conducted at operational airports, safety considerations were paramount. • Personnel were trained and briefed regularly by airport operations personnel. • Although runways were closed, safety areas were identified and drills held so that all assessment personnel were aware of their environment. • Rigorous accounting was made of all experimental FOD taken on the airport.

  10. General Methods • FOD log – Over 1000 items placed and retrieved! • all items were written in a log before proceeding to the runway • after return and often during the testing the log was checked again the items in the box • FOD box – items put in boxes with dividers in order to be easily and quickly counted • Marking • Used for calibration and performance testing • Small, nearly invisible UV marks were placed on the runway • UV lights used to find markings • Locations were surveyed using a differential GPS accurate to millimeters in the X/Y plane • Surveyed locations were compared with locations provided for each detection by each technology

  11. General Methods • Radar • Does NOT depend on illumination • All assessment campaigns at PVD were performed during normal 12 pm to 5 am runway closures • Electro-Optical • Did depend on illumination. • Testing at BOS was on Runway 15R which saw limited use allowing testing at different times. • Tests were performed at BOS during the day, at night, and at dawn and dusk to capture a range of lighting conditions.

  12. Calibration – Standard Target Testing • GOAL = confirm detection performance using objects with defined detection characteristics • Targets selected based on technology • Metal cylinders 3in X 4.5in 1.25in with a nominal reflectivity of 0dBm2 (Large) • Metal cylinders 2in X 2.5in with a nominal reflectivity of -10dBm2 (Medium) • Metal cylinders 1.5in x 1.25in with a nominal reflectivity of -20dBm2 (Small) • Metal 2in spheres • Colored PVC cylinders 1.5in X 1.25in colored grey, white and black

  13. Calibration Targets • Metal cylinders • 1.5in x 1.25in • -20dBm2 • (Small) • Metal cylinders • 2in X 2.5in • -10dBm2 (Medium) • Metal cylinders • 3in X 4.5in 1.25in • 0dBm2 (Large) • Metal spheres 2in Colored PVC cylinders 1.5in X 1.25in

  14. Calibration- Standard Target Testing for TarsierTM • 4 types of metal targets • 6 transects along full length of the runway • Transects 2, 3, and 4 within 1km of both radars • Transects 1, 5, and 6 used for system assessments

  15. Calibration for FODetectTM • Metal cylinders 1.5in x 1.25in tested the radar portion of the sensor • 3 colors of PVC cylinders (white, grey, black) and small metal cylinder • Targets placed at varying distances from sensor in a test rectangle (approx. 150ft x 70ft) • Targets placed to test the system and a single sensor • Standard target array included 1 metal, 1 white, 1 grey, and 1 black cylinder • Groups of targets oriented differently

  16. Performance Testing • Items not selected based on sensor type as with calibration testing • Used examples of common items found at airports • Selection of items based on frequency of occurrence and hazard potential • Calibration items also used during performance testing • Items placed in know positions at specific angles

  17. Table of Standard FOD Items Used in Performance Assessments This table contains the items selected by CEAT

  18. Performance Testing Methods at PVD • 6 arrays related to standard target transects • In array 5x5 grid established, approximately 25ft between items • 23 targets total per array (no items set on runway lights) • Position of items varied between campaigns (random selection of location achieved in test planning); items in the same position in all arrays in any single campaign. • All items rotated 45 degrees after each scan to cover 8 cardinal points of the compass

  19. Performance Testing Methods at BOS • Line established midway between the centerline and the edge line • 5 identical performance items placed at equal distances on this line across the length of test rectangle • Items rotated so detections at 0, 45, and 90 degrees relative center line recorded

  20. Blind Testing Blind Testing Grid at PVD • Used actual FOD items collected from runways • Range of sizes, materials, colors • Over 100 items in collection • 30 items randomly selected for each campaign • Placement locations chosen at random from a grid • Items dropped or tossed so the orientation was random • Up to 10 items used at one time.

  21. Blind Testing Grid of Locations at BOS Typical Grouping of FOD Blind Items

  22. Inclement Weather Testing • Objective to evaluate sensor performance in variable weather conditions • Opportunistic based on long term plan, but scheduled adjusted to coincide with storm events. • Access to the runways in snow was impossible so assessments performed just after snow emergency conditions

  23. Inclement Weather Assessment - Rain • PVD - TarsierTM radar system • October 2007 • Calibration targets were deployed as a rain squall moved across the runway • BOS – FODetectTM hybrid system • March 2008 • One light rain event at dusk

  24. Inclement Weather Assessment - Snow • PVD – TarsierTM radar system • Assessments after 2 snow events • Jan. 24, 2008 – flurries but no accumulation • Feb. 12, 2008 – snow changing to sleet • Time for placement and retrieval were limited due to needs to clear the runway

  25. Inclement Weather Assessment - Snow • BOS – FODetectTM hybrid system • Winter Storm Jan. 7-12 – • Weather Conditions: • rain, freezing rain, freezing drizzle, mist, ice pellets, fog, snow • Totaling 2.1in of wet precipitation • Testing took place on Jan 8, 9, and 13, 2009 • Testing on Jan. 29 and 30 after snow events as well

  26. Summary • 2 systems – radar and hybrid • Assessments performed at PVD and BOS • Assessments took place in variable weather conditions • General protocol – calibration, performance, blind – consistent for all FOD detection system types • Actual procedures were adapted to each senor and adjusted for runway availability • Targets were selected based on • the performance claims of manufacturers • related to sensor characteristics and performance parameters

More Related