Command and Natural Language

1. Command and Natural Language • Command and Natural Languages • Hieroglyphs (a 5000 year old technology) versus icons (the technology of the day) • Special languages • Mathematics • Music • Chemistry • Computers (interactive, user/programming communication)

2. Command and Natural Language • Command and Natural Languages • Goals of a language • Precision • Compactness • Ease of writing and reading • Completeness • Speed in learning • Simplicity in reducing errors • Ease of retention over time

3. Command and Natural Language • Command and Natural Languages • Tools that accelerated language development • Printing Press versus Computer • Print Press provided widespread dissemination of written work • Computer provides widespread dissemination of words via networks • Computers provide tools to manipulate language • Language is a tool to manipulate computers

4. Command and Natural Language • Command and Natural Languages • Computer Languages • Modest influences on spoken natural languages • Huge impact on the development of formal written languages • Early computers were designed to perform mathematical computations • Future is to operate on the real world • Directing robots • Controlling manufacturing • Guiding spacecraft

5. Command and Natural Language • Languages that tell computers what to do

6. Command and Natural Language • Command and Natural Languages • Scripting Languages • HyperCard • Revolution • Database Query Languages • Structured Query Languages (SQL) • The goal of the user is to create a result, not a program (3 to 20 lines) • Boolean operations (AND, OR, NOT) • Command Languages • Ephemeral • Produce an immediate result on an object • Example command line input to a browser: http://www.google.com/ • New command languages have declined as a result of direct manipulation • Typical format: Verb followed by a noun with qualifiers • PRINT MYFILE 3 COPIES • UPD CAD TN=555-555-5555 BILLTN=302-555-1234

7. Command and Natural Language • A UI Design Approach to Command Language Interfaces • Study the user’s task domain to determine the functionality required of the application • Derive a list of task actions and objects • Represent the items as the low-level interface syntax • Update Customer Record • Copy File_A File_B • Determine the frequency of specific tasks and facilitate the execution of those tasks • Determine whether destructive actions are reversible (e.g., delete) • Determine error messages • Transition diagram of how specific commands take the user to specific states

8. Command and Natural Language • Transition Diagrams iNUM oNewNum iCAD iCAD iCAR oNewCar iUPD oNewCAD iCADb iCADa iCPY oNewCPR Logon iCPRa iCPRb oNoCAD iDEL iCAD oNoCPR iCPR

9. Command and Natural Language • UPD CAD=555-555-5555 NUM=444-444-4444 NewNum=333-333-3333 iNUM oNewNum iCAD iCAD iCAR oNewCar iUPD oNewCAD iCADb iCADa iCPY oNewCPR Logon iCPRa iCPRb oNoCAD iDEL iCAD oNoCPR iCPR

10. Command and Natural Language • CPY CPR=555-555-5555 CPR=444-444-4444 iNUM oNewNum iCAD iCAD iCAR oNewCar iUPD oNewCAD iCADb iCADa iCPY oNewCPR Logon iCPRa iCPRb oNoCAD iDEL iCAD oNoCPR iCPR

11. Command and Natural Language • DEL CAD=555-555-5555 iNUM oNewNum iCAD iCAD iCAR oNewCar iUPD oNewCAD iCADb iCADa iCPY oNewCPR Logon iCPRa iCPRb oNoCAD iDEL iCAD oNoCPR iCPR

12. Command and Natural Language • Command Organization • Simplest – each command carries out a single task • Commands plus arguments • COPY FILE_A,FILE_B • DELETE FILE_A • PRINT FILE_A,FILE_B,FILE_C • Delimiters of blanks or commas • Keywords • COPY FROM=FILE_A TO=FILE_B • Options • PRINT/3 PRINTER_LOC_A FILE_A • Command to check seat availability on a flight on August 21 from Washington’s National Airport to LaGuardia Airport departing about 3:00PM • A0821DCALGA0300PM • Parsed: A 0821 DCA LGA 0300PM • Speed

13. Command and Natural Language • Command Organization • Tree Structure • 5x3x4 = 60 tasks based on 5 commands and 1 rule

14. Command and Natural Language • Command Organization • Tree Structure • 5x3x4 = 60 tasks based on 5 commands and 1 rule

15. Command and Natural Language • Command Organization • Tree Structure • 5x3x4 = 60 tasks based on 5 commands and 1 rule

16. Command and Natural Language • Command Organization • Tree Structure • 5x3x4 = 60 tasks based on 5 commands and 1 rule

17. Command and Natural Language • Natural Language Interaction (NLI) • Users communicating in a familiar language (e.g., English) to a computer, and receiving response from the computer in the same language • Potential application - Checkbook maintenance • Pay to University of Delaware on 4/18/06 $1525.00 • Show me all checks made out to the Deer Park • Which checks did I write this week • Previously direct manipulation interfaces have had more success in the market

18. Command and Natural Language • Natural Language Queries (NLQ) • Natural Language front end for databases • The user types an English-like sentence • The result is a structured database search

19. Command and Natural Language • Natural Language Question Answering (NLQA) • “Who was the first Prime Minister of the European Union?” • Don’t attempt an exact answer • Users make incorrect assumptions • There is a president of the European Union • There is a European Commission • Better to return a list of results pages

20. Command and Natural Language • Text-database searching • Used by Lycos, Google, AltaVista • Stemming (attempt to reduce a word to its stem or root form) • Relevance ranking by word-frequency analysis • Filtering of common words (e.g., “the”, “of”, “in”) • Latent semantic indexing • Uses a term-document matrix which describes the occurrences of terms in documents • It is a sparse matrix whose rows correspond to documents and whose columns correspond to terms, typically stemmed words that appear in the documents. • A typical example of the weighting of the elements of the matrix:

21. Command and Natural Language • Natural Language Text Generation (NLTG) • The computer generates readable reports • “White-blood-cell count is 12,000” • “This value exceeds the normal range of 3,000 to 8,000 by 50%” • “Further examination for systemic infection is recommended” • Natural Language, Adventure Games and Instructional Systems • Example user commands • TAKE ALL OF THE KEYS • PICK UP THE SWORD • Instructional software • Guide students through the learning process • Algebra and Geometry • Feedback is provided in conversational language

22. Command and Natural Language • Applications in mobile devices – Voice Commands • Call • Recognizes verb • Accepts name or number as input • Requests clarification by providing a list of names • Confirms dialing • Useful for dialing while driving • Send Text Message • Recognizes verb • Accepts name or number as input • Opens the text composer • Applications in mobile devices – Voice Read Out

23. Command and Natural Language • Natural Language is applicable to situations where the user cannot actually touch the computer or device • Voice input • Synthesized voice output

24. Command and Natural Language • Natural Language Interaction (NLI) vs. Direct Manipulation • Users seek predictable responses • Users get discouraged having to engage in clarification dialogs • Visually oriented interaction exploits • The computer’s capacity for rapid display of information • The human’s ability for rapid visual recognition

25. Command and Natural Language • Natural Language Interaction (NLI) • Turing Test • “Computer program whose conversation is indistinguishable from a human’s” • ELIZA - emulates a Rogerian psychotherapist. • ELIZA has almost no intelligence whatsoever, only tricks like string substitution and canned responses based on keywords. The illusion of intelligence works best, however, if you limit your conversation to talking about yourself and your life. • http://www.manifestation.com/neurotoys/eliza.php3 • AskJeeves • http://www.ask.com/ • Invites natural language questions • Does keyword extraction • Returns web-page result sets

26. Command and Natural Language • Natural Language Interaction (NLI) vs. Direct Manipulation • Users seek predictable responses • Users get discouraged having to engage in clarification dialogs • Visually oriented interaction exploits • The computer’s capacity for rapid display of information • The human’s ability for rapid visual recognition

27. Command and Natural Language • NLI Future Vision – 2001: A Space Odyssey • Astronauts set off in a spaceship controlled by HAL 9000, a revolutionary computer system that is every bit mankind's equal, and perhaps his superior. • HAL endangers the crew's lives for the sake of the mission, and eventually, kills almost all of the astronauts in the film. The astronauts have to overcome the computer. • HAL behaves in the most human fashion of all of the characters. • He reaches human intelligence levels, • Develops human traits of paranoia, jealousy and other emotions. • Humans act like machines • Cooly performing their tasks in a mechanical fashion, • Even under extreme duress. • HAL, develops an acute emotional crisis because he can not accept evidence of his own fallibility.

28. Command and Natural Language • NLI Future Vision – 2001: A Space Odyssey • Check out this video to see the natural communication between the astronauts and the computer. • 2001 Space Odyssey Video • Ease of communication • Personal assessment of Dave’s art work • Determining what is a “personal” question • HALs superior statement of human error to computer error • Preparing the crew psychological report • HAL providing a hard copy of a report • Astronauts talking in private about whether to trust HAL • HAL reading lips http://www.youtube.com/watch?v=Be8Gbqdox68

29. Human Robot Interaction – AI by Spielberg • AI – by Steven Spielberg • Global warming lead humans to near to extinction • Humans create robots known as mechas which can emulate thoughts and emotions. • David is designed to resemble a human child and to virtually feel love for its human owners. • Humans then become extinct • Humanoid’s use David’s memories, to reconstruct the Swinton home • The humanoids also create a clone Monica (his mother). She can only live for one day. • David spends the happiest day of his life playing with Monica and Teddy.

30. Human Robot Interaction – AI by Spielberg http://www.youtube.com/watch?v=xRhCz0ELrKs

31. Human Robot Interaction • More questions and answers • The role of facial expressions? • Context? • Personality? • The role of psychology in applying what is known about behavior to robots? • Education? • Living family trees? http://www.youtube.com/watch?v=pkpWCu1k0ZI http://www.youtube.com/watch?v=IhVu2hxm07E&feature=related

32. Human Robot Interaction http://www.pbs.org/wgbh/nova/tech/social-robots.html

33. Human Robot Interaction • Human-robot interaction (HRI) • The study of interaction dynamics between humans and robots • Robots name first used by Capek’s in the play R.U.R. • http://www.youtube.com/watch?v=mEG7MhltHWk • Robots roles in today’s society • factory automation • service applications • medical care • Entertainment • Asimov in I, Robot (1950) used the term ”robotics” • Three laws of robotics: • A robot may not injure a human being or, through inaction, allow a human being to come to harm. • A robot must obey orders given it by human beings except where such orders would conflict with the First Law. • A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. • http://www.youtube.com/watch?v=WJRfWyTpJRQ&list=PL2FA31FC4B426806D From: Human-Robot Interaction - David Feil-Seifer & Maja J Mataric (2009)

34. Human Robot Interaction • Philip K. Dick’s novel Do Androids Dream of Electric Sheep (1968) • Created replicants are humanoid robots that look and act like humans, • special tests are devised to determine if an individual is a human or a replicant • Test is related to the Turing Test • Asks probing questions that require human experiences and capacities in order to answer correctly. • George Lucas’ Star Wars movies (1977) • C3P0 and R2D2 • http://www.youtube.com/watch?v=1rorneEGPso • Star Trek: The Next Generation (1987-1994), • Data is an android with super-human intelligence but no emotions • http://www.youtube.com/watch?v=qcqIYccgUdM

35. HRI inputs include vision and speech and can… • Process human-oriented data such as facial expression and gestures • Understand language between human users and robots • Apply meaning to the facial and physical expressions and speech • Real-time perceptive abilities of robots • Need to perceive, understand, and react to human activity in real-time • Need to deal with uncertainty in sensing • Depend on specific lighting conditions • Can process physiological signals • Heart rate • Blood pressure • Galvanic skin response (the user’s emotional state) • Mower et al. (predicts when a user is about to quit a rehabilitation-type task) • Body pose and movement (sources of information for social interaction) • Kapoor and Picard implemented an affect recognition system • Detects the attention of human users in order to determine if a user is addressing the robot • Integrates person tracking, face recognition, sound source localization, and leg detection. Human Robot Interaction

36. Embodiment • Impact of physical embodiment on social interaction • Bartneck - the robot was more engaging to the user than a simulated agent • Anthropomorphism • Differences in interaction between anthropomorphic and non-anthropomorphic robots • Children with autism respond to simple mobile car-like robots as well as to humanoid machines. http://www.youtube.com/watch?v=lZSDnvOoX84 • Some studies find that humanoid robots are overwhelming and intimidating • Other studies find that humanoid robots have therapeutic benefit • Biomimetic, (anthropomorphic form) allow human-like gestures and direct imitation movements • Non-biomimetic form preserves the appeal of computers and mechanical objects. • Simplicity/Complexity of Robot Design • Wainer • The more realistic or complex a robot was, the more watchful it seemed • Participants were less likely to share personal information with a realistic or complex robot. Human Robot Interaction

37. Other Attributes • Reeves and Nass • Users experienced a stronger sense of social presence from the agent when the voice type and personality matched, than when they did not • Tapus and Mataric • When a robot’s expressive personality matched the user’s personality, task performance was better than when the personalities were mismatched • Robles used agents that gave feedback for a speed-dating application • Takeuchi • How cultural norms and customs affect the use of robot systems Human Robot Interaction http://www.youtube.com/watch?v=hKAkLUd1jFg

38. Assistive robots • Gives aid or support to a human user • Intended for schools, hospitals, homes • Socially assistive robotics (SAR) • Elder care, education, social and cognitive disorders, and rehabilitation • Post-operative cardiac surgery recovery and Post-stroke rehabilitation • Rehabilitation • Motivating rehabilitation through mutual storytelling • A robot and a user construct a story, which, when acted out, require the user to perform physical therapy exercises. • Feeding, brushing teeth, getting in and out of bed, getting into and out of a wheelchair, and adjusting a bed for maximum comfort • Envision robots as part of a ubiquitous computing system • Combines cameras and other sensors in the environment and computer controlled appliances (such as light switches, doors, and televisions) • The Huggable - a robot outfitted with several sensors to detect different types of touch • http://www.youtube.com/watch?v=nJXkL7bcQR0 Human Robot Interaction

39. Paro - an actuated stuffed seal which behaves in response to touch and sound. • Goal is to provide the benefits of pet-assisted therapy in nursing homes that cannot support pets • Studies have shown lowered stress levels in residents interacting with this robot, as well as an overall increase in the amount of socialization among residents in the common areas of the same facility. • http://www.youtube.com/watch?v=N5WEbw7DOMY • http://www.youtube.com/watch?v=PNw4oicWmWU • Diagnosis and socialization of children with autism spectrum disorders (ASD). • Robots can observe children in ways that humans cannot • Eye-tracking studies have shown remarkable promise when evaluating children for the purposes of diagnosing ASD. • Robots are a more comfortable social partner for children with ASD than people. • These robots encourage social behavior, such as dancing, singing, and playing Human Robot Interaction

40. Human Robot Interaction • Educational Robotics • Goal to develop robot hardware and software to improve education from elementary school onward • Traditional robot competitions • Botball • FIRST • RoboCuphttp://www.youtube.com/watch?v=acrS0kPY-J8 • What need is it filling? • NAO for special education • http://www.youtube.com/watch?v=LoqU0igk_qQ

41. Human Robot Interaction • Ethical Issues for HRI • The technology to support how a robot can make eye contact is comparatively simple • Evaluating how the person is affected by the robot’s gaze and behavior is difficult. • Does the person get bored or frustrated? • Does the user consider the robot helpful and effective? • Is the robot trusted to perform its intended tasks? • Challenges • Unintended uses of the robot • Unintended situations that might be encountered • If the person needs emergency attention, what is the robot’s responsibility?

42. Human Robot Interaction • How safe is the robot itself, and how safe can the robot make life for its user? • The ability to maneuver about a scene without unwanted contact or collisions. • Safety also refers to protection (as much as it is possible) of a robot’s user and of the robot itself. • CASERO- a robot used in a nursing home • http://www.youtube.com/watch?v=dx0zxr3D_zU • Scalability • How well does a robot perform with users from the general population? • How many people can be helped by such a robot • How does speech recognition perform when the speaker has recently suffered a stroke? • Can the robot adapt its behavior to different users?

43. Human Robot Interaction • Social interaction • The primary means of interaction • The driving force behind the design • Autonomy • Autonomy is not desirable in situations such as medication dispersion and therapy monitoring • The user must clearly retain authority • Rehabilitation should terminate if the user is in pain • Imitation • The Total Turing Test • A system communicating in human-like ways (text, speech, facial expressions) tries to fool a human user into believing it is human • A robot’s personality can effect a user’s compliance • When exhibiting a serious personality, the robot can provoke a greater degree of compliance than displaying a playful personality • When the robot’s extroversion/introversion personality traits matched the user’s, task performance was improved • The Uncanny Valley theory suggests that as a robot becomes very similar in appearance to a human, that robot appears less, rather than more, familiar

44. Human Robot Interaction • Privacy • In contrast to ubiquitous systems where a user has no idea of when the system may be watching, robots are tangible and their perception limited and observable. • A robot can be told to leave when privacy is desired, and the user can observe when privacy is achieved. • Because of its synthetic nature, a robot is perceived as less of a privacy invasion than a person, especially in potentially embarrassing situations. • Measuring Social Success • Does the robot successfully achieve the desired social identity? • When the robot is intended to be playful, do users find the robot playful? • If the robot is supposed to be a social peer, do users act as if it were a social peer? • The social identity of the robot (both the personality and the role of the robot) has an effect on the user’s task performance

45. Human Robot Interaction • How the Robot Perceive the Person • Emotion recognition • Vocalizations • Speech • Language • Motor acts • Gestures • GSR • Heart Rate • Radio frequency identification (RFID) tags • Positioned to track and observe children in school hallways to detect when users were in social range, and who they were interacting with over time to help the robot determine appropriate social responses

46. Human Robot Interaction • Nao • http://www.humarobotics.com/en/robotics-lab/nao-humanoid-robot-plays-connect-4/

47. Human Robot Interaction • Facebook Facial Recognition • Security • Or, hold up a photo • Mood Recognition • The API will estimate your mood automatically, with 5 values: Happy, Sad, Surprised, Angry, and Neutral.

48. Human Robot Interaction • Success Relative to Human Caregiver • How does the robot perform relative to a human performing the same task? • In rehabilitation tasks, measure functional improvement • Overall learning measures such as grades, tests, or evaluations can be used. • Spirometry task robot instructs a cardiac surgery patient to do breathing exercises compliance with a human was a suitable can be measured. • A key role of assistive HRI is to provide care where human care is not available. • Cost/Benefit Analysis • Rehabilitation tasks where a robot could serve as a therapist • Giving advice on specific movements • Motivational coach • Monitoring progress • Learning aid, or as a demonstration, showing a user how to do specific exercises. • Low-cost robot vacuum cleaners • Some treated the robot as if it were a member of the household, with status roughly between the vacuum cleaner and a pet. • Others treated it strictly as a device with a purpose. • Does the use of the robot change the cost/benefit ratio of providing such care? • Doe it make such care available where it was not previously possible?

49. Human Robot Interaction • Impact on Caregivers In some cases, the goal is to make the user’s or caregivers’ job easier and more manageable. • By automating the prompting and monitoring of spirometry, which must be performed ten times per hour for the critical post-surgery period, the robot made it possible for caregivers to attend to other tasks and provide more individualized services. • Does the job condition of the caregiver improve as a result of the robot? • How well does the caregiver work with the robot?

50. Human Robot Interaction • Existing Quality of Life Measures • Dementia Care Mapping (DSM) • Robots to observe and measure socially-sensitive care • Eye-contact • Favorable attention • Does the robot result in a general increase in the quality of life as perceived by the user? • Impact on the Role in Community/Society • When fish tanks were introduced into a nursing home environment to test the effects on residents, observers found an overall increase in nutrition on the part of the participating residents • The introduction of new objects of social interest into an environment can thus change the dynamics of the community.