SmartSynth : Synthesizing Smartphone Automation Scripts from Natural Language

1. SmartSynth: Synthesizing Smartphone Automation Scripts from Natural Language Vu Le (UC Davis) Sumit Gulwani (MSR Redmond) Zhendong Su (UC Davis)

2. motivation

3. programming Tasker App Inventor

4. natural language When I receive a new SMS, if the phone is connected to my car’s bluetooth, it reads the message content and replies the sender “I’m driving.”

5. script when(number, content) := MessageReceived() if(IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

6. SmartScript Script P ::= I E T C M Parameter I ::= input (i1, …, in) | ε Event E ::= (r1, …,rn):= whenEvent() | ε Conversions T ::= F1; …; Fn; Condition C ::= if (Π1 ∧ … ∧ Πn) then | ε Body M ::= Stmt1; … ; Stmtn; Conversion F ::= x := Convert(a) Predicate Π ::= Predicate (a1, …, an) Statement Stmt ::= S | foreachx in a doS1; … ; Sn; od Atomic Stmt. S ::= A | F Action A ::= (r1, …, rn):= Action(a1, …, an) Argument a ::= x | i | r | l

7. approach Script = Components + Relations

8. approach Script = Components + Relations

9. approach Script = Components + Relations

10. approach Script = Components + Relations

11. definition • Component • API: MessageReceived • Entity • Literal: “I’m driving” • API return value: MessageReceived.TextO • Relation = <Entity, API parameter> • E.g., < “I’m driving”, SendMessage.TextI>

12. component discovery

13. mapping features • Regular expressions • Bag of words • Phrase length • Punctuation • Parse tree (Stanford NLP parser)

14. relation discovery • Rule-based relation detection • Relative locations of APIs and entities

15. script generation when(number, content) := MessageReceived() if(IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

16. ambiguity

17. incompleteness when(number, content) := MessageReceived() if(IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

18. insight NLP + Program Synthesis

19. ambiguity {MessagedReceived, IsConnectedToBTDevice, Car_BT, Speak, MessageReceived.TextO, (SendMessage OR SendEmail), "I'm driving“ } SendMessage SendEmails Synthesizer Script 1 Script 2 Ranking Script 1 (SendMessage)

20. incompleteness • Search for most likely missing relations

21. q & a • Distinguishing multiple choice questions • Question: API parameter • Answers: Equally-likely related entities What do you want the phone to speak? The received message content “I’m driving”

22. architecture

23. evaluation • 50 scripts collected from user forums • User study: give NL descriptions from scripts • 11 students participated • Each student was given 25 problems • 640 correct descriptions (725 total)

24. component mapping

25. relation detection

26. completing ratio

27. synthesis time

28. related work • General purpose programming using natural language • Natural language interfaces • Specification extraction from natural language

29. thank you NLP Script = Components + Relations Identify Detect Resolve Complete Program Synthesis

30. v.s. Siri Single phone API Composition of APIs Conversation SmartSynth Siri

31. v.s. Tasker