Dr. Randy J. Jost Assistant Professor of Electrical & Computer Engineering Utah State University

1. Applying a Decade of RCS Calibration Activity Experience to the IR/Visible/UV Measurement Community Dr. Randy J. Jost Assistant Professor of Electrical & Computer Engineering Utah State University (435) 797-0789 randy.j.jost@usu.edu Roger W. Davis Consultant, DynCorp/CSC (301) 904-4627 Roger@DavisEmail.us

2. Overview • Introduction • Defining Certification • Key Differences Between RCS Community and EO/IR Community • Creating a Range Book • Report of Measurement / Uncertainty Analysis • Moving Forward as a United Community • How Big a Tent ? • Home – conference • Recommendations • Summary • References

3. Introduction • There is general agreement that the EO/IR has need of a mechanism to improve it ability to: • Compare data between sensors, systems and programs • Understand the quality of measurements and errors • Gain consensus in a very diverse community • Key question is “How do we get there from here?” • The purpose of this presentation is to raise awareness of some of the key issues, engender discussion, and suggest a starting point for addressing the concerns of the community members.

4. Certification - Why? • When you are up to your rear in alligators, it’s hard to remember your objective was to drain the swamp (moat) or …. • Certification helps you to achieve the desired end result by understanding and documenting the “measurement alligators” a.k.a error sources. Non Sequitur by Wiley, Published in the Washington Post, 15 Jan 1997

5. Certification - What? • Objective: Uniformly improve signature measurements across the EO/IR spectrum by all “measurement facilities” for all sponsors • Method: Establish ISO-9000 Quality System within the EO/IR measurement community • How: Standardize and document measurement processes & procedures

6. Certification - What? • A Quality System: • Within a Formal Structure, establish: • Defined processes & responsibilities • A means for evaluating effectiveness • A means for making improvements • Based on a Cultural Mindset that includes: • All activities involved in generating the product* or service must be controlled to some extent, ideally in a systematic manner • Every person & action affecting the quality of the product or service is a part of the Quality System * Product = EO/IR Measurement Data Set(s)

7. Certification - What? Phases of a ISO-9000 Quality System • Preparation • Educating, Assessing, Planning, Scheduling • Documentation • Define policies, objectives • Develop/document procedures & work instructions • Implementation • Operate as documented, internal audits, corrective & preventive actions, management reviews • External Assessment • External evaluation & registration are optional • Surveillance & Improvement • On-going internal audits, continuous improvements

8. Certification - What? ISO XX Compliance Level • Key ISO Definitions: • ISO-XX “Registered”: A company or sub-unit attests that it has met the relevant criteria for ISO-XX as determined by an internal “self inspection” review. (Sometimes called “self” certification) • ISO-XX “Certified”: A company or sub-unit who is ISO-XX “Registered” is reviewed by a third party of peers (from other companies or other divisions) and that third party attests that the company does indeed meet the relevant criteria for ISO-XX • ISO-XX “Accredited” : Company is reviewed by an independent organization/firm/company who specializes in ISO-XX compliance. (I.E. reviewers are full-time specialists). Company is “accredited” if this firm attests they meet the relevant criteria for ISO-XX. Lowest Medium Highest

9. Certification - What? ISO XX Compliance Level • Key ISO Definitions: • ISO-XX “Registered”: A company or sub-unit attests that it has met the relevant criteria for ISO-XX as determined by an internal “self inspection” review. (Sometimes called “self” certification) • ISO-XX “Certified”: A company or sub-unit who is ISO-XX “Registered” is reviewed by a third party of peers (from other companies or other divisions) and that third party attests that the company does indeed meet the relevant criteria for ISO-XX • ISO-XX “Accredited” : Company is reviewed by an independent organization/firm/company who specializes in ISO-XX compliance. (I.E. reviewers are full-time specialists). Company is “accredited” if this firm attests they meet the relevant criteria for ISO-XX. Lowest Medium Highest

10. Certification - What? Why Certified instead of Registered or Accredited? • Registered : Incurs 80+% of sunk cost and effort • Certification: Most of the Technical Benefit comes from the third party peer review • Different perspective offered by reviewers • Reviewers have a wealth of experience to share (add value) • If you can’t explain a process properly to a reviewer, chances are the process needs clarification (feedback). • If reviewer(s) rate a practice “unsatisfactory” they are required to provide written comments so range becomes satisfactory! (Goal: Pass, not fail) • Certification Effort protects proprietary / competitive interests • ANSI-Z-540 review process is confidential…only the final certification compliance report is available to customers • “Corrective actions” need not be shared once implemented (Embarrassment factor is understood and respected)

11. Certification - What? The term “CERTIFICATION,” as used in the DoD demonstration program and RCS community (in the sense of a facility being “certified”), is defined as having successfully completed a detailed review, of an RCS Range Book1 by a peer review committee using published evaluation criteria2. 1 Written in accordance with the quality documentation standard: “ RCC/SMSG Report 804-01” 2 Evaluation Criteria are in the appendix of RCC Report 804-01 For more information on the history and details of the DoD RCS Range Certification Demonstration program, see the paper “Lessons Learned in Applying ANSI/NCSL Z-540 to the RCS Community and Transferring Those Lessons to the IR Measurement Community,” by Davis/Jost on the CalCon CD provided with the registration material.

12. Certification - What? • Detailed review and compliance assessment that a range has met the relevant quality assurance documentation standard for ANSI Z-540 • Certification is: • A Review committee’s positive assessment of compliance with the ANSI-Z-540 standard • An endorsement of the range’s measurement processes • “Deliverable” Documentation Package is the Range Book • Certification isNOT: • A rating system • A review committee endorsement of day-to-day quantitative capabilities (accuracy, uncertainty, etc)

13. Key Differences Between RCS and EO/IR Communities • The IR community is a much more segmented group than the RCS community, and thus faces additional challenges and advantages. • Government is not the only driver, but still a big player. • Because of the large commercial component, community consensus is even more important.

14. Key Differences Between RCS and EO/IR Communities • Challenges • More types of measurements => more types of calibration approaches. • reflection, emission, transmission and absorption processes. • Incoherent, partially coherent, coherent radiation • Spectrum of interest is much more “diverse” than that of the RCS community. • Environment has a much bigger impact on measurement and resultant uncertainty.

15. Key Differences Between RCS and EO/IR Communities • Range book sections will have to be tailored for many more different types of measurements: • Active vs. Passive • Spectral vs. temporal • Imaging vs. non-imaging • Visual vs. IR vs. UV vs. Laser • Single band / Broadband vs. Multi-band vs. Hyper-spectral • Component vs. subsystem vs. system

16. Building a Range book for the EO/IR Community RCS community has developed the following structure, following the ANSI Z540 “outline”: 1 Introduction and Endorsement 2 References 3 Glossary 4 Organization and Management 5 Quality System, Audit, and Review 6 Personnel 7 Accommodation and Environment 8 Equipment and Reference Materials 9 Measurement Traceability and Calibration 10 Calibration Methods 11 Handling of Calibration Items 12 Records 13 Certificates and Reports 14 Subcontracting of Calibration 15 Outside Support Services and Suppliers 16 Complaints 17 Inter-range Comparison Programs 18 Data Processing Procedures 19 Range-Specific Uncertainty Analysis 20 Ongoing Research, Planned Improvements

17. Building a Range book for the EO/IR Community 1 Introduction and Endorsement 2 References 3 Glossary 4 Organization and Management 5 Quality System, Audit, and Review 6 Personnel 7 Accommodation and Environment 8 Equipment and Reference Materials 9 Measurement Traceability and Calibration 10 Calibration Methods 11 Handling of Calibration Items 12 Records 13 Certificates and Reports 14 Subcontracting of Calibration 15 Outside Support Services and Suppliers 16 Complaints 17 Inter-range Comparison Programs 18 Data Processing Procedures 19 Range-Specific Uncertainty Analysis 20 Ongoing Research, Planned Improvements • Priority One – Direct Impact on Data Quality • Sections 9, 10, 11, 17, 18, 19

18. Building a Range book for the EO/IR Community 1 Introduction and Endorsement 2 References 3 Glossary 4 Organization and Management 5 Quality System, Audit, and Review 6 Personnel 7 Accommodation and Environment 8 Equipment and Reference Materials 9 Measurement Traceability and Calibration 10 Calibration Methods 11 Handling of Calibration Items 12 Records 13 Certificates and Reports 14 Subcontracting of Calibration 15 Outside Support Services and Suppliers 16 Complaints 17 Inter-range Comparison Programs 18 Data Processing Procedures 19 Range-Specific Uncertainty Analysis 20 Ongoing Research, Planned Improvements • Priority One – Direct Impact on Data Quality • Sections 9, 10, 11, 17, 18, 19 • Priority Two – Enabling Data Quality • Sections 5, 8, 13, 20

19. Building a Range book for the EO/IR Community 1 Introduction and Endorsement 2 References 3 Glossary 4 Organization and Management 5 Quality System, Audit, and Review 6 Personnel 7 Accommodation and Environment 8 Equipment and Reference Materials 9 Measurement Traceability and Calibration 10 Calibration Methods 11 Handling of Calibration Items 12 Records 13 Certificates and Reports 14 Subcontracting of Calibration 15 Outside Support Services and Suppliers 16 Complaints 17 Inter-range Comparison Programs 18 Data Processing Procedures 19 Range-Specific Uncertainty Analysis 20 Ongoing Research, Planned Improvements • Priority One – Direct Impact on Data Quality • Sections 9, 10, 11, 17, 18, 19 • Priority Two – Enabling Data Quality • Sections 5, 8, 13, 20 • Priority Three – Supporting Documentation • Sections 1, 2, 3, 4, 6, 7, 12, 14, 15, 16

20. Building a Range book for the EO/IR Community 1 Introduction and Endorsement 2 References 3 Glossary - Terminology 4 Organization and Management 5 Quality System, Audit, and Review 6 Personnel 7 Accommodation and Environment 8 Equipment and Reference Materials 9 Measurement Traceability and Calibration 10 Calibration Methods 11 Handling of Calibration Items 12 Records 13 Certificates and Reports 14 Subcontracting of Calibration 15 Outside Support Services and Suppliers 16 Complaints 17 Inter-range Comparison Programs 18 Data Processing Procedures 19 Range-Specific Uncertainty Analysis 20 Ongoing Research, Planned Improvements • Priority One – Direct Impact on Data Quality • Sections 3, 9, 10, 11, 17, 18, 19, 20 • Priority Two – Enabling Data Quality • Sections 5, 8, 13 • Priority Three – Supporting Documentation • Sections 1, 2, 4, 6, 7, 12, 14, 15, 16

21. NIST Handbook 152 Recommended Practice; Symbols, Terms, Units and Uncertainty Analysis for Radiometric Sensor Calibration • Appendix 2 is a copy of NIST TN1297, 1994 Edition. • This should be the starting point for developing range book content. • Supports both Section 3 and Uncertainty Analysis

22. Report of Measurement /Uncertainty Analysis • Next to capturing the details of the measurement process (and the activities that affect the data quality), this is probably the key portion of the range book. • If the community does nothing else but determine an accepted, common format for an Uncertainty Analysis or Report of Measurement, it will be a huge step forward. • Effort put forth to accomplish this will provide immediate payoff. • Will provide a mechanism to make quantifiable comparisons between data sets or measurements from different programs, sensors, etc.

23. Report of Measurement Contents • Identify target being calibrated • Identify parameter(s) being determined [Major activity of this Workshop] • Briefly describe and reference the measurement procedures • Specify parameter values AND uncertainty • Tabulate sources of error AND uncertainty for each source • At least reference how each uncertainty value was obtained • At least state how individual source uncertainties were combined to obtain overall uncertainty • Clearly state the type of measurement the report of measurement is reporting on, and the frequency range/band the measurements are good for

24. Moving Forward as a Community • Who are the Players? • Government – DoD, NASA, NOAA, NIST, etc. • Industry – Government Contractors, Private Companies • Academia – Universities, Research Labs • Who becomes the Prime Mover? • Government entity • Community consensus

25. Moving Forward as a Community • Where to talk about this effort? • SPIE/ASPRS/ISPRS – Lost in the crowd? SPIE

26. Moving Forward as a Community • Where to talk about this effort? • Military Sensing Symposia • Caters to DoD • Membership • Classification issues

27. Moving Forward as a Community • Where to talk about this effort? • Calcon • Favors remote sensing community, but open to other users • Covers the breadth of optical, IR and UV calibration issues

28. Annual Conference on Characterization & Radiometric Calibration for Remote Sensing: • Addresses characterization, calibration, and radiometric issues within the IR, Visible and UV spectrums. • Session Topics Include: • Concepts and Applications of Measurement Uncertainty • Solar, Lunar and Stellar Radiometric Measurements • Pre-launch to On-orbit Calibration Transfer: Approaches and On-orbit Monitoring Techniques • Developing National Calibration/Certification Standards for EO/IR Systems Join us at Utah State University September 13-16, 2004, For the 13th Annual CalCon

29. Recommendations • Develop a consensus on where the “community” wants to go • Find a forum for like-minded players to use as a focal point for developing the measurement improvement program • Develop a common terminology • Develop a taxonomy of measurement “categories” • Develop Uncertainty Analysis / Report of Measurement template • Conduct education process with customers and end users

30. Summary Range certification is worth doing. It IS a lot of work, and does require resources, but … experience has taught that benefits will outweigh the costs. Community: needs to decide what it wants to do, to what extent and with whom. First task is to start speaking the same language and partitioning the problem into manageable “chunks.”

31. Summary Measurement Facility: Starting point is documenting (in detail) what you do and how you do it. Range book is the best way to store your process documents and supporting documentation and make them available to your customers (internal and external). Report of Measurement (RoM) with uncertainty analysis is a key document. Tailor contents of Range book, RoM, to your needs while maintaining consistent structure of documentation across ranges.

32. References ISO 17025 Home Page: http://www.fasor.com/iso25/ NIST Tech Note 1297 – Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results. Online version at: http://physics.nist.gov/Pubs/guidelines/contents.html or a PDF version is at http://physics.nist.gov/Documentation/tn1297.pdf Uncertainty of Measurements Home Page: http://physics.nist.gov/cuu/Uncertainty/index.html Signature Measurements Home Page: http://www.signaturemeasurements.org/ “Air Force Research Laboratory Advanced Compact Range RCS Uncertainty Analysis of a General Target,” B.M. Welsh, W.D. Muller, and B.M. Kent, APS Magazine, June 2003, pp. 195-200. Antenna Measurement Techniques Association (AMTA) Annual Symposium Proceedings for past several years.

33. Concluding Thought "If you can't describe what you are doing as a process, you don't know what you're doing." - W. Edwards Deming “If you can’t document your process, no one else will believe you know what you are doing.” - R. Jost

34. Backup Slides For more information

35. What drives the error / uncertainty? Source: Wyatt, “Electro-optical System Design For Information Processing, McGraw-Hill, 1991.

36. What drives the error / uncertainty? “Easy” to characterize Accuracy – “High” Uncertainty – “Low” Source: Wyatt, “Electro-optical System Design For Information Processing, McGraw-Hill, 1991.

37. What drives the error / uncertainty? “Hard” to characterize Accuracy – “Low” Uncertainty – “High” Source: Wyatt, “Electro-optical System Design For Information Processing, McGraw-Hill, 1991.

38. ANSI/NCSL Z540-1-1994 • ANSI version of ISO 25, now superceded by ISO 17025. • Starting point for the RCS community

39. Handbook for the Interpretation and Application of ANSI/NCSL Z540-1-1994 • Contains interpretive information for implementing Z540-1 • Useful, but community needs to write its own version tailored to its needs.

40. ANSI/NCSL Z540-2-1997 • This is the ANSI version of “The Guide to the Expression of Uncertainty in Measurement. • Very good reference for developing an Uncertainty Analysis or Report of Measurement.

41. NIST Technical Note 1297, 1994 Edition Meant to capture NIST’s policy for including uncertainty values in NIST measurements and reports.

42. Why Facility Tailoring is the Keyor Consequences of Not Using Common SenseorThe Dangers of Extrapolation In the space of one hundred and seventy-six years, the Lower Mississippi has shortened itself two hundred and forty-two miles. That is an average of a trifle over one mile and a third per year. Therefore, any calm person, who is not blind or idiotic, can see that in the Old Oolitic Silurian Period, just a million years ago next November, the Lower Mississippi River was upward of one million three hundred thousand miles long, and stuck out over the Gulf of Mexico like a fishing rod. And by the same token any person can see that seven hundred and forty-two years from now the Lower Mississippi will be only a mile and three-quarters long, and Cairo and New Orleans will have joined their streets together, and be plodding comfortably along under a single mayor and a mutual board of aldermen. There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact. Mark Twain, Life on the Mississippi (1883).

43. For More Information • For a copy of an example RCS range book, contact: • Chris Clark • Mission Research Corporation • 3975 Research Boulevard • Dayton, OH 45430-2108 • (937) 253-8379 • Email: cclark@mrcday.com • For on-line resource material: • Signature Measurements Home Page: • http://www.signaturemeasurements.org/