Instilling reliability in Application to identify Silicon Disk Surface Containments

1. Instilling reliability in Application to identify Silicon Disk Surface Containments EMIS 7305 Project Due: 4/28/2011 Stephen Tinh (31060345)

2. Agenda • Overview • Analysis / Study • Results • Summary / Conclusion

3. Overview • Examine how reliability can be built into a software application project • Software does not suffer failure from the same external conditions that plague hardware components (shock, temperature, fatigue, etc…) • Undertaken as a desire to relate assignment to my personal field as a junior software systems engineer • Working on program in maintenance phase with a role in implementing fixes or new enhancements through the use of software and firmware only • Software Application will identify silicon surface deformities from an electron beam microscope image • Project idea drawn from a friend who is working on this application as a personal education project/challenge

4. Starting Assumptions • Provided sample image from electron beam microscope is 8-bit grayscale image (0-255 shades of gray). • The center of the defect is present within approximately the center 10x10 pixels of the image. • The defect is illustrated in a reasonable size within the image (i.e., the defect isn’t zoomed beyond the edge of the image). • Image size does not exceed a resolution of 1024 pixels in width and 954 pixels in height. • Image only contains 1 instance of 1 type of copper ball, pit, or protrusion deformity.

5. Analysis • Discovered that software reliability is much harder to quantify when compared to hardware reliability • Subject is seldom mentioned in traditional reliability texts (absent in MIL standards 470B/785A, and articles) until recently • Definition of software reliability is the measure of a system to operate without failure when in service • Since software always executes as written (not assuming AI!), therefore, reliability of software is directly related to how well the software product addresses the need • Many plans for software reliability promote well defined processes to (1) ensure software operates as intended and (2) contains no inadvertent faults • Item (1) is generally obtainable, item (2) usually warrants the most attention • Infinitely many practices available to grow software reliability, we tailored the process to pick and choose elements which we found useful in our situation

6. Analysis • Investigating the start of the design phase, a design methodology should be selected to provide a development format and structure • Water fall, Incremental, Spiral, etc… • We selected the Evolutionary Model, which works best when, • Development teams are small • Change is occurring rapidly • Intermittent build evaluation help feedback into design process • We worked all the design tasks concurrently • Noted that this model does not provide a rigid structure for process repeatability (again okay in our case!)

7. Analysis • All software design begins with a need statement and outlined software requirements • Can draw from feasibility studies, group elicitation, similar programs, etc… • One of the best methods I discovered for building software requirements (especially for GUI based applications) is through Use Cases • Use cases provide driven scenarios on the interaction between the user and the application • Technique also models variant branches of what-if examples to help flush out performance requirements • Executed this exercise through brainstorming activities overlunch

8. Analysis • Context, data flow, and state diagrams are stated as crucial in software development • Illustrate relationship and traffic flow of elements in the design • I believe the main strength is its ability to clearly define inputs and outputs of functions or objects • Before coding began, we generated a context illustration which contained projected functions and a top level view of the expected traffic • Due to the compact size of the application, we felt data flow or timing diagrams weren’t warranted

9. Analysis • Software reuse is a quick and easy way to automatically build reliability • Can be thought of as obtaining COTS products in hardware design • Reuse operations written by subject matter experts, accredited/compliant software, or those generally accepted by members in your field – and saves time! • Our software application borrows from a few Matlab toolkits and libraries recommended by the Matlab Online Community for 2D image processing

10. Analysis • In software development, you need to be able to track baselines and the incorporated changes of each build • Since this was a recommended software design practice, we selected an open-source configuration management tool called Mercurial • In our small application, we spent more time setting it up than utilizing it as we only had 2 iterations of software builds to capture • However, for mission critical or larger software efforts, change management would be a requirement (especially important in the Evolutionary Design Model we selected at the beginning)

11. Analysis • Testing is essential in the design of quality software • The goal is to identify unintended behaviors or faults before the software is released • It can also provide design feedback that can be flowed back into the design for the next iteration • We generated Software Test Descriptions to verify that the software is meeting the outlined software requirements we planned • It also contains a traceability map to the requirement each step is testing, verification method, and the affected Use Cases

12. Analysis • The 3 surface defects we are concerned with identifying are pictured in sample shots below,

13. Results • We generated a set of 12 requirements that captures the essential functions of the application (continually changing as needed through evolutionary model) • Select examples,

14. Results • Example snippet, • Use cases heavily influenced our design of the GUI interface • We brainstormed through each button as a separate use case • Generated Use Cases for • Launch GUI and 1 variant path • Load Image Button and 2 variant paths • Process Image Button and 2 variant paths

15. Results • Our context diagram partitions the software into 4 main parts- the GUI interface, the Main function, Process image function, and the determine defect function • Also, reveals our top level expectations for the inputs and outputs of each element

16. Results • The software does image processing on the sample images before passing it to the resolution function that borrows from a few recommended Matlab toolkits to determine the shape and identify the defect • Unfortunately, at the deadline of this project, outputting the location of the defect [R-6] was not ready (although by design it always has to appear near the middle of image anyways) • The GUI was created per the requirements with the minimum load, process, and quit user buttons *not representative of final implementation, GUI and sample shots were taken from intermediate builds

17. Results • A method called Statistical Testing was used to track performance of the intermediate builds • Only 2 major iterations were required to achieve the desired accuracy in our defined test of sample images • The defined test of sample images to be loaded 1 by 1 and evaluated, included, • 28 copper ball test shots • 3 pit test shots • 9 protrusion test shots • 1 scrambled image (our out of bounds metric) • The first software build performed with the following results,

18. Results • An evaluation was performed to understand the performance of the application in reference to the results (good and bad) • The copper ball had no faults due to the high delta of contrast between the ball and the surface silicon that allows it to be isolated and recognized from the background scene of the image. • The pit defect appeared to consistently be detected, but upon closer inspection, one sample was inadvertently correct due to how the logic tree worked. This type of unintended behavior serves as an example of a recommend fault to address in the subsequent build, however the developer chose not to focus his attention on this item for the time being. • The protrusion detection exhibited the worse success rate due to a few of the image samples resembling a copper ball defect (even to a human observer). This will have to be addressed in the next iteration in order to grow towards our accuracy goal. • The scrambled image correctly returned inconclusive. • To address our findings, the next iteration included a change for clarifying protrusion detection that considers the raised surface of the silicon that is exhibited in the protrusion case, but not the copper ball case (where the silicon surface remains flat). • Re-running the same experiment with each image again, we obtained the following results

19. Results • Although in our situation, it only took the developer 2 iterations to achieve an acceptable accuracy, I believe this still provides rationale in illustrating that reliability grown in software can be obtained through frequent testing, and analysis • The results can be funneled back into the development process to make better design decisions • Similar to the FMECA analysis in RCM mentioned by the guest lecturer in class • Implementing Statistical Testing on a larger scale, the curve for the number of faults would ideally resemble, • where after each test and analysis iteration, the number of failures decreases towards the acceptable limit

20. Results • The Software Test Description covers the steps for verifying the software meets the outlined requirements • A sample snippet of the case we generated shown below (also containing verification method and use case traceability)

21. Summary / Conclusion • Unlike hardware reliability which is quantifiable and objective, I learned software reliability is a more subjective art • Software is indifferent to environmental conditions such as temperature or humidity, and it does not wear-out with usage • Thus, a different methodology must be used when trying to achieve software reliability • This primarily comes in the form of implementing software reliability techniques that span the development lifecycle aimed at producing fault-free operating software • This project exposed me to a side of reliability that isn’t yet as well defined or covered as hardware reliability, so I consider it an excellent learning experience and relatable