Considerations In Designing a Conformity Assessment System for Voting Equipment Presented at

1. Considerations In Designing a Conformity Assessment System for Voting Equipment Presented at The Voting Systems Testing SummitNovember 28-29, 2005Sacramento, CA

2. Purpose of Presentation • This presentation discusses: • Elements common to conformity assessment systems • The EAC system for certification and decertification of voting equipment • Considerations for state and local programs • Potential for unintended consequences

3. Certification of a product is a means of providing assurance that it complies with specified standards and other normative documents. Several types of certification systems exist and are under review for the EAC system: • Some comprise type testing only • Other systems include initial testing and field surveillance • Still others include initial testing of a product and assessment of its suppliers’ quality systems, followed by routine audits that take into account the factory quality system and the testing of samples from the factory and the open market.

4. Components of a Certification System • Initial Type Testing • Assessment of the Supplier’s Quality System • Field Information & Feedback • User Involvement

5. Key Participants • EAC • State certification authorities • Testing Laboratories • Vendors • State & local election officials

6. Key Issues for Certification System • What is a minimal acceptable system? • Are the testing lab/testers/lab assessors qualified? • Will the vendor deliver units within manufacturing tolerances to those tested? • How will the election officials know if non-compliant units are delivered and what corrective actions can it take? • Will election officials and poll workers use the systems as intended?

9. Coordinated Processes • Processes are necessary to implement the components of a conformity assessment system for voting systems and to provide reasonable assurance that the key concerns of such a system are carefully addressed. • The best confidence will be gained if the various authorities involved in certifying and selecting voting equipment coordinate their efforts so as to maximize the value contributed from each function.

10. EAC / NISTTGDC Technical Guidelines Candidate System Accredited Test Laboratory EAC Technical Review Team Certified System System Description NIST Software Ref. Lib. Vendor Quality & ConfigurationControl Processes State Certification

11. Certified System System Description NIST Software Ref. Lib. Vendor Quality & ConfigurationControl Processes State Certification Purchase Selection Purchase Selection Field Deploymentand Use Pre & Post Election Testing

12. EAC / NISTTGDC Technical Guidelines EAC Technical Review Team Accredited Test Laboratory NIST NVLAP&Software Ref. Lib. State & LocalElection Officials Vendors Local Receiving Inspectors Citizens

13. System Description The ability to determine that a system is identical to that considered at other steps in the process is critical to the validity of the conformity assessment system.

14. Typical FCC Submission The FCC routinely requires external AND internal photos accompany all submissions. For Software Controlled Radios source code is required for an equipment grant.

15. Typical FCC Submission External Photos

16. Typical FCC Submission Internal Photos

17. Challenge for Vendors Is there a way by which high confidence many be delivered that a system is exactly the same without also transmitting proprietary information?

18. Considerations How does each step of the process assist the next? Could the ITA’s include a section in each test report highlight information and observations that would be helpful to state certification personnel? Could state certifiers transmit observations to those making purchasing decisions?

19. Considerations What is the repeatability of these evaluations? How often will the same system receive the same evaluation if submitted at different time or to different ITA’s?

20. Considerations How does field performance and field experience information come back into the system to improve it? How will field experience come into the system to improve EAC and state certification processes?

21. Unintended Consequences Cost of certification will limit the number of systems offered and may eliminate vendors. With more rigorous testing and evaluation vendors will pay for fewer systems to be certified. Vendors will bring new systems to market less often. Some vendors may abandon this business.

22. Unintended Consequences Will the time and expense of certifying changes and upgrades delay good and needed changes? What is the right balance between careful evaluation of all changes and not making these evaluations so costly that needed improvements are delayed or not introduced at all?

23. Unintended Consequences How will the necessary coordination of administrative processes and equipment requirements be maintained and evaluated? Election management and equipment requirements must work together. How will this be assured? Where several alternative solutions are possible how will the linkage be maintained between equipment requirements and election management practices?

24. Proposals Documents are needed that define the interaction and expectations of each participant in the process. Direct communications from each step in the process to succeeding steps may be very helpful. Clarification is needed as to what is not done in preceding steps in the process.

25. Conclusion • A well constructed certification system provides satisfactory answers to central issues: • What is a minimal acceptable system? • Are the testing lab/testers/lab assessors qualified? • Will the vendor deliver units within manufacturing tolerances to those tested? • Will the election officials know if non-compliant units are delivered and what actions can they take? • Will election officials and poll workers use the systems as intended?