Download
1 / 48
E N D
1. Spoken Programs or Programmers say the darndest things! Andrew Begel, Susan L. Graham
Computer Science Division, EECS
University of California, Berkeley
2. VL/HCC 2005 Motivation Programmers conventionally use keyboard
Long hours at keyboard leads to higher risk of RSI
Can a programmer code using speech?
Can a computer understand what the developer says? If the user can say anything, this makes it really difficult to understand.
Provide motivations for doing this work.
Learning about more natural modes of interaction
Put why programming by voice on this slide.
Introduce programming by voice as a solution.
We use conventionalized programmign languages bcause of training and tools.
Not the best way to do things, but too hard to change, so instead we’ll modify the
Input to the programming language to make it easier for people.
(programming ambiguous because it makes the compielr and validation easier).
Already existing tools for adapting to programming languages (declaration before def)
Given the programming domain
Here are the problems we have with them.
If the user can say anything, this makes it really difficult to understand.
Provide motivations for doing this work.
Learning about more natural modes of interaction
Put why programming by voice on this slide.
Introduce programming by voice as a solution.
We use conventionalized programmign languages bcause of training and tools.
Not the best way to do things, but too hard to change, so instead we’ll modify the
Input to the programming language to make it easier for people.
(programming ambiguous because it makes the compielr and validation easier).
Already existing tools for adapting to programming languages (declaration before def)
Given the programming domain
Here are the problems we have with them.
3. VL/HCC 2005 Programming by Voice Our Goal
Find out how developers use code verbally. Use this to develop a naturally verbalizable input form.
Build development environment that supports verbal authoring, navigation, modification.
Extend conventional compiler analyses to support ambiguities generated by speech.
Learn how developers can use voice-based programming, and iterate design. speak whole story, then say we’re going to talk about the first one.
What we want to do: comp, nav, edit
speak whole story, then say we’re going to talk about the first one.
What we want to do: comp, nav, edit
4. VL/HCC 2005 Programming by Voice Challenges Speech is inherently ambiguous.
Programming languages and tools were designed to be unambiguous.
Speech tools are poorly suited for programming tasks.
Programmers are not used to verbal software development. We use conventionalized programmign languages bcause of training and tools.
Already existing tools for adapting to programming languages (declaration before def)
(programming not ambiguous because it makes the compiler and validation easier).
Introduce programming by voice as a solution.
Not the best way to do things, but too hard to change, so instead we’ll modify the input to the programming language to make it easier for people.We use conventionalized programmign languages bcause of training and tools.
Already existing tools for adapting to programming languages (declaration before def)
(programming not ambiguous because it makes the compiler and validation easier).
Introduce programming by voice as a solution.
Not the best way to do things, but too hard to change, so instead we’ll modify the input to the programming language to make it easier for people.
5. VL/HCC 2005 Programming by Voice for (int i = 0; i < 10; i++ ) {
¦
}
6. VL/HCC 2005 Current Tools are Awkward! for (¦ ; ; ) {
} Existing tools badly adapt VR to programming
(No compiler technology)
More explicitly list out loud what the awkwardness is.
Requires users to spell identifiers
VR finds small words hard to understand.
prescriptive way to enter the program
once code is in editor, it can no longer be edited by template.
code entry is over-stylized
no solution for nav and editing
Existing tools badly adapt VR to programming
(No compiler technology)
More explicitly list out loud what the awkwardness is.
Requires users to spell identifiers
VR finds small words hard to understand.
prescriptive way to enter the program
once code is in editor, it can no longer be edited by template.
code entry is over-stylized
no solution for nav and editing
7. VL/HCC 2005 for (int i = 0;¦ ; ) {
} More explicitly list out loud what the awkwardness is.
More explicitly list out loud what the awkwardness is.
8. VL/HCC 2005 for (int i = 0; i < 10;¦ ) {
} More explicitly list out loud what the awkwardness is.
More explicitly list out loud what the awkwardness is.
9. VL/HCC 2005 for (int i = 0; i < 10; i++ ) {
¦
} More explicitly list out loud what the awkwardness is.
not the first person to worry about this stuff, here’s what people have to live with now. More explicitly list out loud what the awkwardness is.
not the first person to worry about this stuff, here’s what people have to live with now.
10. VL/HCC 2005 How do Programmers Speak Code? 10 programmers read Java code out loud (Begel ‘05)
Graduate students in Computer Science
Five knew Java, five did not
Five were native English speakers, five were not
Five were educated in U.S.A., five were not
Read pre-written code into tape recorder
As if speaking to a sophomore-level CS undergrad who knows Java, but does not know the program
Most programmers spoke the same way
11. VL/HCC 2005 for (int i = 0; i < 10; i++ ) {
¦
} Natural Language Verbalization Didn’t have to verbalize parens, semicolons, etc.
Didn’t have to verbalize parens, semicolons, etc.
12. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
13. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
14. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
15. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
16. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
17. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
18. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
19. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
20. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
21. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
22. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
23. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
24. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
25. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
26. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
27. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
28. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
29. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
30. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
31. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
32. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
33. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
34. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
35. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
36. VL/HCC 2005 How Do Programmers Speak Code? “Think about how to say this”
“Think about how to say this”
37. VL/HCC 2005 Design Tradeoffs
38. VL/HCC 2005 Spoken Java Semantically identical to Java
Syntactically easier to say than Java
Design methodology generalizable to any computer language
39. VL/HCC 2005 A More Natural Way to Code public class Symbol implements Serializable {
¦
} Didn’t have to verbalize parens, semicolons, etc.
Didn’t have to verbalize parens, semicolons, etc.
40. VL/HCC 2005 A More Natural Way to Code public class Symbol implements Serializable {
static HashMap hashtbl = new HashMap(); ¦
} All punctuation has English equivalents
Open Brace, End For Loop
Most punctuation is optional
Provide verbalization for all abbreviations
Relaxed phrasing for better fit with English
(int)foo ? “cast foo to integer”
foo = 6 ? “set foo to 6”
foo[i]++ ? “increment the ith element of array foo”
All punctuation has English equivalents
Open Brace, End For Loop
Most punctuation is optional
Provide verbalization for all abbreviations
Relaxed phrasing for better fit with English
(int)foo ? “cast foo to integer”
foo = 6 ? “set foo to 6”
foo[i]++ ? “increment the ith element of array foo”
41. VL/HCC 2005 A More Natural Way to Code public class Symbol implements Serializable {
static HashMap hashtbl = new HashMap();
}
¦ All punctuation has English equivalents
Open Brace, End For Loop
Most punctuation is optional
Provide verbalization for all abbreviations
Relaxed phrasing for better fit with English
(int)foo ? “cast foo to integer”
foo = 6 ? “set foo to 6”
foo[i]++ ? “increment the ith element of array foo”
All punctuation has English equivalents
Open Brace, End For Loop
Most punctuation is optional
Provide verbalization for all abbreviations
Relaxed phrasing for better fit with English
(int)foo ? “cast foo to integer”
foo = 6 ? “set foo to 6”
foo[i]++ ? “increment the ith element of array foo”
42. VL/HCC 2005 A More Natural Way to Code for (int i = 0; i < 10; i++ ) {
¦
} Didn’t have to verbalize parens, semicolons, etc.
Didn’t have to verbalize parens, semicolons, etc.
43. VL/HCC 2005 Too Many Ambiguities for (int i = 0; i < 10; i++ ) {
¦
} A human could eyeball this.A human could eyeball this.
44. VL/HCC 2005 Sometimes It’s Non-Obvious fore *= 8; file.tooLode.times = won ¦
45. VL/HCC 2005 SPEED: Speech Editor Build an editor that supports naturally verbalized programs
SPEED: SPEech EDitor
Based on Dragon Naturally Speaking, Eclipse IDE, Harmonia
Spoken Java Language for Composition
Spoken Command language for Navigation, Editing, Template instantiation, Refactorings, Search
Visual feedback for training
46. VL/HCC 2005 Future Study - SPEED Usability Goal: Understand how SPEED can be used by expert programmers
Train expert Java programmers on SPEED
Author new code
Build a Linked List data structure with associated algorithms
Modify existing code
Change abstraction representation and update algorithms Say this is for expert java programmers.
Rephrase system understanding (everything else is about the person, but this is about the system).
Find some terminology from AI systems evaluations (false positive rate etc)
Say this is for expert java programmers.
Rephrase system understanding (everything else is about the person, but this is about the system).
Find some terminology from AI systems evaluations (false positive rate etc)
47. VL/HCC 2005 Metrics User Metrics
Task completion time
Vocabulary mistakes, Forgotten vocabulary
Grammatical mistakes, Grammatical substitutions
SPEED Metrics
Word tokenization errors
Disambiguation errors
Irresolvable ambiguities
48. VL/HCC 2005 Spoken Programs Learn how developers speak code out loud
Analyze speech and categorize ambiguities
Use study to design naturally verbalizable input for programming
Create programming tools designed for spoken interaction
Evaluate design and tools by studying programmers using voice for software development fix animation
fix animation
49. VL/HCC 2005 End of Talk
