Lee T. Ostrom, Cheryl Wilhelmsen and Roger Scott

1. Lee T. Ostrom, Cheryl Wilhelmsen and Roger Scott Development of a Risk Assessment Framework

2. Overview • Project Objectives • Project Team • Year One– Identification of Operational Factors • Year Two – Visual Inspection Experiment • Year Three – Risk Framework Development

3. Project team: Lee T. Ostrom, Ph.D. Cheryl Wilhelmsen, Ph.D. Roger Scott, Ph.D. Project Team

4. Year One – Operational Factors • We collected data from: • Foreign MRO • Domestic MRO • Domestic Regional MRO • Military composite shop

5. Year Two • During Year Two the goal was to determine how well inspectors and AMTs could visually inspect composites. • Included in this was to determine to what extent OFs impact the inspection process.

6. Year Three During Year Three we analyzed the data from the first two years of the project We combined the data and developed several basic error rates associated with inspection of composite materials We also developed an overarching risk framework

7. Risk Framework Definition • A bounded set of activities associated with the visual inspection of composite materials that can generate risk, • the ways in which these activities can fail, • the failure/error rates for these activities, • the major influences, for instance PSFs, on these activities that can increase or reduce the risk, • and the potential ways in which the risk can be mitigated or eliminated.

8. Reason for an Inspection We found four basic reasons a visual inspection might be performed. These are: Formal scheduled inspection – these inspections are mandated from the maintenance manuals for the aircraft. These can be GVI or DVI inspections Walk-around inspections – these inspections occur when a pilot or some other airline personnel notice an anomaly on an aircraft and notifies maintenance who then will send a mechanic or inspector to look at it. These inspections most likely start as a GVI and then can progress to a DVI.

9. Reason for an Inspection Event driven inspections – these inspection occur due to something like a bird strike, lighting strike or hail damage. These inspections most likely start as a GVI and then can progress to a DVI. Informal inspections – similar to walk-around inspections, these inspections occur when a ramp person or other airline notice an anomaly and notifies maintenance who then will send a mechanic or inspector to look at it. These inspections most likely start as a GVI and then can progress to a DVI.

10. Type of Inspections • We were only concerned with the two types of visual inspection: • GVI • DVI

11. Type of Inspections • It is important to note that most inspectors don’t purely do just a GVI • If they see an issue they will also many times tap test the area with whatever tool they normally use • Coin/token • Flashlight • Tap hammer

12. Inspection Process We developed a very simple model of the visual inspection process to work from in regards to developing our risk framework This model is shown on the next page

13. Formal Scheduled Inspection DVI Inspection Walkaround Inspections NDT/NDE Inspection Event Driven Inspections GVI Inspection Informal Inspection Inspection Space

14. Risk of Inspection Though there are areas of risk associated with each step of the process we chose to focus on the inspection process itself

15. Inspection Event During the act of inspection humans both collect and analyze data. We recognize that data collection and analysis cannot be entirely separated during the inspection event. Data analysis involves inspectors’ perception as to whether or not the anomalies represent damage requiring repair, or further analysis using NDE/NDT techniques.

16. Inspection Event Both data collection and data analysis are influenced by the process by which anomalies are identified; the characteristics of the anomalies that are identified, and the organizational system in which the processes reside

17. What is Damage? One of the big questions in the Year-Two experiments was: What is Composite Damage?

18. Corrosion around rivet heads These dents around rivets could appear to an inspector as if they were damage and there is some corrosion, but it is not composite damage per se.

19. Crack Crack

20. Probability of Detection Curves

21. Probability of Detection Curves

22. Error Types The error types we focused on were associated with the inspection processes From the experimental data we developed some nominal error/failure rates associated with the error types

23. Error Rates

24. Event Tree Development Inspection Initiated GVI Inspection DVI Inspection End State Inspector fails to perceives surface anomaly as damage and there is no damage Safe Inspector fails to perceives surface anomaly as damage and there is damage Damage is undetected Inspector fails to perceives surface anomaly as damage and there is no damage Inspector perceives surface anomaly as damage and there is no damage Safe Inspector perceives surface anomaly as damage and there is no damage Inspector perceives surface anomaly as damage and there is no damage False Positive Inspector perceives surface anomaly as damage and there is damage Inspector perceives no damage, but there is damage Damage is undetected Inspector perceives damage NDT/NDE performed or part is repaired/replaced

25. Fault Tree for the Inspection Process Undetected Damage OR AND Inspection isn’t Performed AND Surface anomaly is actual damage Inspector fails to perceive anomaly as damage using DVI Inspector fails to perceive anomaly as damage using GVI Surface anomaly is actual damage

26. Human Reliability Event Trees a. Inspector perceives paint chip as damage using GVI. A. Inspector fails to perceive paint chip as damage using GVI. B. Inspector fails to perceive paint chip as damage using DVI b. Inspector perceives paint chip as damage using DVI. c. NDT/NDE inspector confirms paint chip is damage. C. NDT/NDE inspector fails to confirm paint chip is damage.

27. Risk Assessment for Bottom-Side of Test Article

28. Risk Assessment for Bottom-Side of Test Article

29. Risk Assessment for Bottom-Side of Test Article

30. Risk Assessment for Bottom-Side of Test Article

31. Risk Assessment for Bottom-Side of Test Article Undetected Damage OR Failure to Detect Crack on B1 Failure to Detect Large Hole in B2 Failure to Detect Dent on B3 Failure to Detect Paint Chip on D5 Failure to Correctly Categorize Dent on E2 Failure to Detect Dent on E3 Falsely Identify Quadrants as Having Damage

32. Risk Assessment for Bottom-Side of Test Article

33. Failure Rate Comparison

34. Factors Affecting Inspection Errors • The condition of the inspector. For instance: • Fatigue • Eyesight • Environmental conditions: • Lighting • Access • Noise

35. Factors Affecting Inspection Errors • Although, in many cases the factor does not prevent the inspector from performing the task • For instance, lighting. In the case of lighting the inspector always carries a high intensity flashlight to compensate for bad lighting

36. Factors Affecting Inspection Errors • Tap testing is another factor that is performed in a number of different ways and the ambient noise might impact the performance of this test • However, in many cases the inspectors wait until times when the hanger is quieter before they do tap testing

38. Summary Findings • We feel there are three (2) primary takeaways from the research we conducted. These are: • GVI is error prone. Because GVI only requires visual observation of the composite surfaces there is a much higher potential for missing true damage and for indicating false positives

39. Summary Findings • In real life inspections it appears that GVI is never truly used by itself. • When we observed inspectors in the field it was always indicated that if they saw a surface anomaly they then used some other technique, such as touching the surface or tapped the surface to get a better understanding as to whether it was true damage or not.

40. Summary Findings • DVI on the other hand has a second check built into it. Inspectors usually look at a surface and then touch or tap it with a tap hammer or coin/token to get a better understanding as to whether it is damaged.  

41. Summary Findings 2. The more sensory input the inspectors can bring to bear on the inspection task, the more likelihood of a positive outcome. • That is, the inspectors identified fewer false positive detections of damage, and the additional sensory input produced results more consistent with actual damage.

42. Remote Inspection Using Three Dimensional Imaging Technology

43. Remote Inspection Using 3D Imaging The idea behind this is that a base scan is created using LIDAR and then during subsequent years LIDAR scans will be performed and compared to see if damage has occurred. Software tools had been developed to contrast the two scans and the contrasted images show graphically the changes over time.

44. Inspection Proof of Concept • To test this: • We LIDARed our test article • We inflicted some additional damage on the test article • We LIDARed the part again • The Center for Advanced Energy Studies (CAES) technician performed the contrasting routine • We then used the CAES Computer Assisted Virtual Environment — or CAVE TM to determine whether we could see the inflicted damage.

45. Hole we saw from a prior slide Dent we also inflicted

46. Results We could see all the damage we inflicted on the part. The contrasting software also provided a very visual indication of damage.

47. Results It is clearly apparent that software could be developed that could semi-automate this type of inspection. Also, using the 3D environment an inspector 10,000 miles or more away could perform a relatively detailed visual inspection without being in physical proximity to the aircraft.

48. Remote Inspection Using 3D Imaging Update We have received a small grant to advance this work We will scan either a small composite aircraft or a NASA UAV

49. Remote Inspection Using 3D Imaging Update We will simulate damage on the aircraft and LIDAR it again We will use the contrasting software to determine whether the simulated damage can be detected by the software and visually in the CAES visualization CAVE

50. Our Concept UAV or Conventional Aircraft Flying Along Minding its Own Business