Interactive Robot Theatre: Integration of Machine Learning, Quantum Networks, and Software-Hardware Methodology in Human

1. Interactive Robot Theatre as a future toy Integration of Machine Learning, Quantum Networks and software-hardware methodology in humanoid robots Marek Perkowski, Dept. Electrical Engineering PSU, and Department of Electronics and Computer Science, Korea Advanced Institute of Science and Technology Talk presented at Department of Electronics, Technical University of Warsaw, December 2004

2. Toys is a very serious business

3. Talking Robots • Many talking toys exist, but they are still very primitive • Actors for robot theatre, agents for advertisement, education and entertainment. • Designing inexpensive natural size humanoid caricature and realistic robot heads Dog.com from Japan We concentrate on Machine Learning techniques used to teach robots behaviors, natural language dialogs and facial gestures. Work in progress

4. Robot with a Personality? • Future robots will interact closely with non-sophisticated users, children and elderly, so the question arises, how they should look like? • If human face for a robot, then what kind of a face? • Handsome or average, realistic or simplified, normal size or enlarged? • The famous example of a robot head • is Kismet from MIT. • Why is Kismet so successful? • We believe that a robot that will interact with humans should have some kind of “personality” and Kismet so far is the only robot with “personality”.

5. Robot face should be friendly and funny The Muppets of Jim Henson are hard to match examples of puppet artistry and animation perfection. We are interested in robot’s personality as expressed by its: • behavior, • facial gestures, • emotions, • learned speech patterns.

6. Behavior, Dialog and Learning • Robot activity as a mapping of the sensed environment and internal states to behaviors and new internal states (emotions, energy levels, etc). • Our goal is to uniformly integrate verbal and non-verbal robot behaviors. • Words communicate only about 35 % of the information transmitted from a sender to a receiver in a human-to-human communication. • The remaining information is included in para-language. • Emotions, thoughts, decision and intentions of a speaker can be recognized earlier than they are verbalized. NASA

8. Morita’s Theory

10. Robot Head Construction, 1999 Furby head with new control Jonas We animate various kinds of humanoid heads with from 4 to 20 DOF, looking for comical and entertaining values.

11. Mister Butcher Latex skin from Hollywood 4 degree of freedom neck

12. Robot Head Construction, 2000 Skeleton Alien We use inexpensive servos from Hitec and Futaba, plastic, playwood and aluminum. The robots are either PC-interfaced, use simple micro-controllers such as Basic Stamp, or are radio controlled from a PC or by the user.

13. Technical Construction, 2001 Details Marvin the Crazy Robot Adam

14. 2001 Virginia Woolf heads equipped with microphones, USB cameras, sonars and CDS light sensors

15. 2002 BUG (Big Ugly Robot) Max Image processing and pattern recognition uses software developed at PSU, CMU and Intel (public domain software available on WWW). Software is in Visual C++, Visual Basic, Lisp and Prolog.

16. Visual Feedback and Learning based on Constructive Induction 2002

17. 2002, Japan Professor Perky Professor Perky with automated speech recognition (ASR) and text-to-speech (TTS) capabilities • We compared several commercial speech systems from Microsoft, Sensory and Fonix. • Based on experiences in highly noisy environments and with a variety of speakers, we selected Fonix for both ASR and TTS for Professor Perky and Maria robots. • We use microphone array from Andrea Electronics. 1 dollar latex skin from China

18. Maria, 2002/2003 20 DOF

19. location of head servos Construction details of Maria skull • location of controlling rods • location of remote servos Custom designed skin

20. Animation of eyes and eyelids

21. Software/Hardware Architecture • Network- 10 processors, ultimately 100 processors. • Robotics Processors. ACS 16 • Speech cards on Intel grant • More cameras • Tracking in all robots. • Robotic languages – Alice and Cyc-like technologies.

22. Cynthia, 2004, June

23. Currently the hands are not moveable. We have a separate hand design project.

24. HAHOE KAIST ROBOT THEATRE, KOREA, SUMMER 2004 Sonbi, the Confucian Scholar Paekchong, the bad butcher

25. Yangban the Aristocrat and Pune his concubine The Narrator

26. The Narrator

28. We base all our robots on inexpensive radio-controlled servo technology.

29. We are familiar with latex and polyester technologies for faces

30. New Silicone Skins

31. Probabilistic State Machines to describe emotions “you are beautiful” / ”Thanks for a compliment” P=1 Happy state “you are blonde!” / ”I am not an idiot” P=0.3 “you are blonde!” / Do you suggest I am an idiot?” Unhappy state P=0.7 Ironic state

32. Facial Behaviors of Maria Do I look like younger than twenty three? Maria asks: Response: • “no” • “no” • “yes” 0.7 0.3 Maria smiles Maria frowns

33. Probabilistic Grammars for performances Speak ”Professor Perky”, blinks eyes twice P=0.1 Speak ”Professor Perky” Where? P=0.3 Who? P=0.5 P=0.5 P=0.5 Speak “in some location”, smiles broadly Speak “In the classroom”, shakes head Speak ”Doctor Lee” What? P=0.1 P=0.1 P=0.1 Speak “Was singing and dancing” P=0.1 Speak “Was drinking wine” ….

34. Human-controlled modes of dialog/interaction Human teaches “Thanks, I have a lesson” “Hello Maria” “Lesson finished” Robot performs Robot asks “Question” “Stop performance” “Questioning finished” “Command finished” “Thanks, I have a question” “Thanks, I have a command” Human asks Human commands

35. Robot-Receptionist Initiated Conversation Human Robot What can I do for you? Robot asks This represents operation mode

36. Robot-Receptionist Initiated Conversation Human Robot What can I do for you? I would like to order a table for two Robot asks

37. Robot-Receptionist Initiated Conversation Human Robot Smoking or non-smoking? Robot asks

38. Robot-Receptionist Initiated Conversation Human Robot Smoking or non-smoking? I do not understand Robot asks

39. Robot-Receptionist Initiated Conversation Human Robot Do you want a table in a smoking or non-smoking section of the restaurant? Non-smoking section is near the terrace. Robot asks

40. Robot-Receptionist Initiated Conversation Human Robot Do you want a table in a smoking or non-smoking section of the restaurant? Non-smoking section is near the terrace. A table near the terrace, please Robot asks

41. Human-Initiated Conversation Human Robot Hello Maria initialization Robot asks

42. Human-Initiated Conversation Human Robot Hello Maria What can I do for you? Robot asks

43. Human-Asking Human Robot Question Question Robot asks Human asks

44. Human-Asking Human Robot Yes, you ask a question. Question Human asks

45. Human-Asking Human Robot Yes, you ask a question. What book wrote Lee? Human asks

46. Human-Asking Human Robot I have no sure information. What book wrote Lee? Human asks

47. Human-Asking Human Robot I have no sure information. Try to guess. Human asks

48. Human-Asking Human Robot Lee wrote book “Flowers”. Try to guess. Human asks

49. Human-Asking Human Robot Lee wrote book “Flowers”. This is not true. Human asks

50. Human ends questioning Human-Teaching Human Robot Questioning finished Human asks Human teaches “Questioning finished” Robot asks Thanks, I have a lesson