The Case Of The Killer Robot - Praxis

1. The Case Of The Killer Robot -Praxis �The Smoking Gun� Mike Wicks 8th November 2005

3. Current Situation Randy Samuels is indicted on charges of manslaughter Samuels was formerly employed as a programmer at Silicon Techtronics Inc. The charge involves the death of Bart Matthews, who was killed last May by an assembly-line robot Matthews worked as a robot operator at Cybernetics Inc. He was crushed to death when the robot he was operating malfunctioned. The robot arm struck Matthews, throwing him against a wall and crushing his skull. Matthews died almost instantly The indictment Samuels wrote the particular piece of computer program responsible for the robot malfunction

4. The Scenario (1) �Silicon Valley Programmer Indicted for Manslaughter� (newspaper article) Story line - Randy Samuels, programmer for Silicon Techtronics, is indicted for manslaughter in the death of Bart Matthews, robot operator Ethical issues - Should a programmer be held legally responsible if his or her code causes bodily injury or death to a human being? �Developers of �Killer Robot� Worked Under Enormous Stress� (newspaper article) Story line - The killer robot project was already far behind schedule when management hired twenty new programmers to speed things up. The project leader, Ray Johnson, expounds his �Ivory Snow Theory� of software reliability: 99.44/100% pure is pure enough Technical issues - human relationships, conflicting software philosophies, mythical man-month, just how reliable does software have to be? Ethical issues - Competence, Implications of abusive verbal behaviour

5. The Scenario (2) ��Killer Robot� Programmer was Prima Donna, Co-Workers Claim� (newspaper article) Story line - A software psychologist analyses Randy Samuels� personality and discovers that he is task-oriented, like everyone else on his team. Various team members are interviewed to get a more complete picture of his personality and of the team dynamics on the team Technical issues - Programmer psychology, personality types, egoless programming, team organisations, properties of successful versus unsuccessful teams Ethical issues - Implication of abusive verbal behaviour ��Killer Robot� Project Mired in Controversy Right from Start� (newspaper article) Story line - A dispute arose on project concerning the use of waterfall model versus prototyping Technical issues - software process models, waterfall model, prototyping model, role of institutional inertia in decision making Ethical issues - Awareness of one�s own technical limitations, is it ethical to ignore the end-user in product development?

6. The Scenario (3) �Silicon Techtronics Promised to Deliver a Safe Robot� (newspaper article) Story line - A group of programmers come to Randy Samuels� defence by making public the requirements document that served as a legal contract between Silicon Techtronics and Cybernetics, Inc. (the customer) Technical issues - Nature of a requirements document, functional and non-functional requirements, user and operator training Ethical issues - Duties and obligations with respect to a document of this nature �The �Killer Robot� Interface� (paper in academic journal, although not a good one) Story line - Professor Gritty argues that the design of the user interface was ultimately responsible for the disaster. He explains Shneiderman�s Eight Golden Rules [5] in some detail and shows how the killer robot interface violated each and every rule Technical issues - user interfaces and software quality, Shneiderman�s Eight Golden rules, computers in workplace and ergonomics Ethical issues - Computers in the workplace

7. The Scenario (4) �Software Engineer Challenges Authenticity of �Killer Robot� Software Tests� (newspaper article) Story line - Wesley Silber, Professor of Software Engineering, discovers a discrepancy between documented software tests and the robot�s actual behaviour. In so doing, he explains the nature of software testing Technical issues - Software testing, white box vs. black box testing, static vs. dynamic testing, unit vs. integration testing, version management Ethical issues - Just how reliable does software have to be? �Silicon Techtronics Employee Admits Faking Software Tests� (newspaper article) Story line - Cindy Yardley is exposed as the software tester that faked the software tests, acting at the request of her boss, Ray Johnson. These and other goings on at Silicon Techtronics are exposed by Max Worthington, corporate security officer, who had the job of monitoring employee e-mail Technical issues - Just how reliable does software have to be? Ethical issues - whistle blowing, just how reliable does software have to be, data privacy, software theft, faking software tests, codes of ethics

8. The Scenario (5) �A Conversation with Dr. Harry Yoder� (Sunday newspaper magazine article / interview) Story line - Dr. Yoder presents his thesis that a corporation is a social organism, and as such, the leaders of that organism bear a special responsibility for what has transpired Technical issues - Corporation as organism Ethical issues - Techniques for making ethical decisions, legal vs. ethical, accountability and responsibility in software failures

9. Who or What �pulled the trigger�? Where does �the Buck Stop�? Vendor? People Methods Development Methods Culture Purchaser? People Methods Physical Realisation

10. Vendor - People Management Ray Johnson (Robotics Division Head) Development Team Sam Reynolds (CX30 Project Manager) Individual Developers Randy Samuels (Programmer) Quality Assurors Cindy Yardley (Software Tester)

11. Vendor - Methods Management Methods Development Methods Programming Methods Programming Language Quality Assurance

12. Vendor - Development Methods Analysis Specification Design Implementation Testing

13. Vendor - Culture Leadership Team Spirit Pride in Job �Safety First� �Ivory Snow� Pressure

14. Purchaser - People Management Staff Trainers Installation Engineers Machine Operators Maintenance Personnel

15. Purchaser - Methods Training Regimes Safety Practices Installation Procedures Live Testing Maintenance Procedures

16. Purchaser - Physical Realisation Installation Environment Hardware Configuration Safety Mechanisms

17. So, who was responsible for the death of Bart Matthews?

18. Questions How much input should a User have into the process of designing a computer system that they are going to use? What caused the robot accident? Who are the suspects? Under what conditions would YOU be comfortable with the Robbie CX30 returning to service? Note the differences between "accidental death by firearm" and "accidental death by poorly programmed robot". Should the law make such a distinction? Should such laws stress punishment or prevention? What is the difference between Civil and Criminal Liability ?

19. Other Cases You all might not work for a robot manufacturer in the end, however, can you think of other cases when bad code could cause harm to people or things?

20. What would I do? Informal guidelines: Is there something you or others would prefer to keep quiet? Does it pass the Mom Test? Would you tell her? Would she do it? Does it pass the TV Test? Would you tell a nationwide audience? Does it pass the Market Test? Could you advertise the policy to gain a market edge? Does your instinct tell you something is wrong? Does it pass the Smell Test Does the situation "smell"? Ernest A. Kallman and John P. Grillo. "Solving Ethical Dilemmas: A Sample Case Exercise." Ethical Decision Making and Information Technology. New York: McGraw-Hill, 1996: pp33-56.

21. Further Questions Central to Killer Robot Story, is the idea that new media have had a dramatic impact and the full impact, good and bad has not been fully realised. Many issues in the tale fall artfully short of a conclusion and force the reader to ask questions such as: Who should be held responsible for software that malfunctions and causes harm? Is faulty different from malicious? Does High technology dull our sensitivity to the real world? What "Human" skills (like conversation) are being lost because of computer technologies? What prevents artificial intelligence from become artificial stupidity? Should there be a new authority to handle legal problems from these new technologies? Would there be one for each country?

22. Trials - Personnel In Pairs of Groups A vs B, C vs D, E vs F, G vs H 1st group (A, C, E & G) provides: Prosecution Lawyer Judge Prosecution Witnesses (all remaining group members) 2nd group (B, D, F & H) provides: Defence Lawyer Defendant Defence Witnesses (all remaining group members) Audience is the Jury

23. Trials - Running Order Opening Speeches 1 min max. each Prosecution first, then defence Witnesses 6 mins total each No cross examination Prosecution first, then defence Closing Speeches 1 min max. each Prosecution first, then defence Judge�s Summing Up 1 min max. Jury Vote 2 mins max. Judge�s sentencing and/or comments 1 min max.

24. Trials � Different Defendants A vs B Randy Samuels CX30 Programmer @ ST C vs D Ray Johnson Robotics Division Chief @ ST E vs F Cindy Yardley CX30 Software Tester @ ST G vs H Sam Reynolds CX30 Project Manager @ ST

25. Trials This weeks tutorial Start preparation of case Groups A & B � Mike Nelson in EM 3.36 Groups C & D � Mike Wicks in EM 3.02 Groups E & F � Lisha Ma in EM 1.70 Groups G & H � Brain Palmer in EM G.44 Following 2 weeks tutorials Trials delivered A vs B, D vs E Week 7 in room EM 3.36 C vs F, G vs H Week 8 in room EM 3.36 Group Mark awarded (10% of final mark) Absent with no good reason � 0 marks!

26. Portfolio Assignment #1 At least 2 Feedback sheets from peers Author Reflection Sheet Assignment #2 (DRAFT) At least 2 Feedback sheets from peers Author Reflection Sheet Assignment #2 (FINAL) PDP Signed by your Mentor Deadline 12 noon on Friday 11th November 2005, (Week 6 - this week!) Place in box in crush area Fill in Blue Cover sheet Sign it Label as Praxis (no tutor name necessary)

27. 5 Minutes to think and chat Any questions about: The format of the cases? What you need to do? What you have to hand in? Not who I think was responsible!!

28. Acknowledgments Euan Dempster