scaffolding instructions to learn procedures from users l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
SCAFFOLDING INSTRUCTIONS TO LEARN PROCEDURES FROM USERS PowerPoint Presentation
Download Presentation
SCAFFOLDING INSTRUCTIONS TO LEARN PROCEDURES FROM USERS

Loading in 2 Seconds...

play fullscreen
1 / 13

SCAFFOLDING INSTRUCTIONS TO LEARN PROCEDURES FROM USERS - PowerPoint PPT Presentation


  • 479 Views
  • Uploaded on

Paul Groth and Yolanda Gil Information Sciences Institute University of Southern California SCAFFOLDING INSTRUCTIONS TO LEARN PROCEDURES FROM USERS Learning Procedures Naturally Humans learn procedures using a variety mechanisms Observation, practice, reading textbooks

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'SCAFFOLDING INSTRUCTIONS TO LEARN PROCEDURES FROM USERS' - lotus


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
scaffolding instructions to learn procedures from users
Paul Groth and Yolanda Gil

Information Sciences Institute

University of Southern California

SCAFFOLDING INSTRUCTIONS TO LEARN PROCEDURES FROM USERS
learning procedures naturally
Learning Procedures Naturally
  • Humans learn procedures using a variety mechanisms
    • Observation, practice, reading textbooks
  • Human tutorial instruction
    • Broad descriptions of actions and explanations of their dependencies
  • The computer is told what to do by the instructor.
  • Goal: learn procedures from instruction that is natural to provide
what is instruction by telling
What is Instruction by Telling
  • General statements
    • Not refer to a specific state
  • Descriptive statements
  • About types, functions, processes

“To dial a number, lift the receiver and punch the number”

“Place a pot with water on the stove. Turn the stove on and wait until the water bubbles”

“A good hovering area is behind a tall building that is more than 200 ft away”

“A vehicle is parked if it is stopped on the side of the road or if it is stopped in a parking lot for more than 3 minutes”

why is instruction by telling important
Why is Instruction by Telling Important?
  • [Scaffidi et al 06]: by 2012, 90M end user programmers in the US alone
    • 13M would describe themselves as programmers
    • 55M will use spreadsheets and databases
  • [Adams 08]: we have gone from dozens of markets of millions of users to millions of markets of dozens of users
    • The “long tail of programming” [Anderson 08]
  • Today most successful end user applications focus on data manipulation through spreadsheets and web forms
  • We need approaches to allow end users to specify procedures to process data or to control a physical environment
    • With examples
    • By telling: a natural method for humans, needed if procedures are complex and hard to generalize from examples
shortcomings of human instruction gil 09
Shortcomings of Human Instruction [Gil 09]
  • Organization
  • Omissions
  • Structure
  • Errors
  • Student’s preparation
  • Student’s ability
  • Teacher’s skills

“[…] Procedures are complex relational structures and the mapping between these structures and a linear sequence of propositions expressed in discourse is not easy to define.” – [Donin et al 02]

tellme learning procedures by being told
TellMe Learning Procedures by Being Told
  • Developed four-stage process:
    • Ingestion: create initial procedure stub from given instruction
    • Elaboration: map terms to existing knowledge, infer missing information using heuristics, create hypotheses of procedures
    • Elimination: rule out hypotheses through symbolic execution
    • Selection: select one procedure hypothesis using heuristics that maximize consistency
an example
An Example

Which player?

Closest could have meant closest sideline

or it could have meant closest player

Instruction is ambiguous &

incomplete

What is “closest”?

Repeat until when?

<j.0:Procedure rdf:about="#SetupKlondikeSolitaire">

<j.0:hasSteps rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

<j.0:Loop>

<j.0:until rdf:resource="#Unknown>

<j.0:repeat rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Decrement"/>

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

</j.0:repeat>

</j.0:Loop>

</j.0:hasSteps>

<j.0:hasInput rdf:resource="#deck"/>

<j.0:hasResult rdf:resource="#solitaireGameSetup"/>

</j.0:Procedure>

2. start SetupKlondikeSolitaire hasId 8887

3. resultIsname=solitaireGameSetup isatype=GameSetup

4. initSetupname=deck isatype=CardDeck

5. doThisname=Deal basedOn deck, 7 expectname=hand

6. name=hand isatype=Hand

7. doThisname=Layout basedOn hand

8. name=numOfCards isatype=Integer

9. name=numOfCards value=7

10. repeatdoThisname=Decrement basedOn numOfCards expect numOfCards

Create initial

interpretations based on prior

knowledge, annotate gaps

1

<j.0:Procedure rdf:about="#SetupKlondikeSolitaire">

<j.0:hasSteps rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

<j.0:Loop>

<j.0:until rdf:resource="#Unknown>

<j.0:repeat rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Decrement"/>

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

</j.0:repeat>

</j.0:Loop>

</j.0:hasSteps>

<j.0:hasInput rdf:resource="#deck"/>

<j.0:hasResult rdf:resource="#solitaireGameSetup"/>

</j.0:Procedure>

Repeat until left sideline is reached

or right sideline or front line

Elaborate using

heuristics for filling gaps

2

Repeat until teammate

or opponent has ball

<j.0:Procedure rdf:about="#SetupKlondikeSolitaire">

<j.0:hasSteps rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

<j.0:Loop>

<j.0:until rdf:resource="#Unknown>

<j.0:repeat rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Decrement"/>

<j.0:Procedure rdf:about="#Deal"/>

< <j.0:Procedure rdf:about="#Deal"/ j.0:Procedure rdf:about="#Layout"/>

</j.0:repeat>

</j.0:Loop>

</j.0:hasSteps>

<j.0:hasInput rdf:resource="#deck"/>

<j.0:hasResult rdf:resource="#solitaireGameSetup"/>

</j.0:Procedure>

<j.0:Procedure rdf:about="#SetupKlondikeSolitaire">

<j.0:hasSteps rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

<j.0:Loop>

<j.0:until rdf:resource="#Unknown>

<j.0:repeat rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Decrement"/>

<j.0:Procedure rdf:about="#Deal"/>

< <j.0:Procedure rdf:about="#Deal"/ j.0:Procedure rdf:about="#Layout"/>

</j.0:repeat>

</j.0:Loop>

</j.0:hasSteps>

<j.0:hasInput rdf:resource="#deck"/>

<j.0:hasResult rdf:resource="#solitaireGameSetup"/>

</j.0:Procedure>

<j.0:Procedure rdf:about="#SetupKlondikeSolitaire">

<j.0:hasSteps rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

<j.0:Loop>

<j.0:until rdf:resource="#Unknown>

<j.0:repeat rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Decrement"/>

<j.0:Procedure rdf:about="#Deal"/>

< <j.0:Procedure rdf:about="#Deal"/ j.0:Procedure rdf:about="#Layout"/>

</j.0:repeat>

</j.0:Loop>

</j.0:hasSteps>

<j.0:hasInput rdf:resource="#deck"/>

<j.0:hasResult rdf:resource="#solitaireGameSetup"/>

</j.0:Procedure>

<j.0:Procedure rdf:about="#SetupKlondikeSolitaire">

<j.0:hasSteps rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

<j.0:Loop>

<j.0:until rdf:resource="#Unknown>

<j.0:repeat rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Decrement"/>

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

</j.0:repeat>

</j.0:Loop>

</j.0:hasSteps>

<j.0:hasInput rdf:resource="#deck"/>

<j.0:hasResult rdf:resource="#solitaireGameSetup"/>

</j.0:Procedure>

<j.0:Procedure rdf:about="#SetupKlondikeSolitaire">

<j.0:hasSteps rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

<j.0:Loop>

<j.0:until rdf:resource="#Unknown>

<j.0:repeat rdf:parseType="Collection">

<j.0:Procedure rdf:about="#Decrement"/>

<j.0:Procedure rdf:about="#Deal"/>

<j.0:Procedure rdf:about="#Layout"/>

</j.0:repeat>

</j.0:Loop>

</j.0:hasSteps>

<j.0:hasInput rdf:resource="#deck"/>

<j.0:hasResult rdf:resource="#solitaireGameSetup"/>

</j.0:Procedure>

X

X

X

Eliminate through

symbolic execution and

reasoning

3

Select based on heuristics

that maximize consistency

4

example heuristics
Example Heuristics
  • Ingestion
    • If a variable is assigned a constant in the instruction, then find a consistent basic type for it.
  • Elaboration
    • If the input of a component (i.e. subtask) is type compatible with the result of a preceding component, then that result could be connected to the input.
    • If two variables share any typing information they could be unified.
  • Elimination
    • Hypotheses with matching symbolic execution traces can considered to be the same.
  • Selection
    • Pick the simplest hypothesis (with least components).
instructions that tellme can process
Instructions that TellMe Can Process

// Stop and move back to your previous position

(e.g. cut back).

11: doThis name=MoveTowards

12: basedOn originalPosition expect=currentPosition

// If you are not open, do this again.

13: until

14: name=Open basedOn currentPosition

// Once your open, find the ball and face it.

15: doThis name=FindTheBall expect=ballLocation

16: name=ballLocation isa type=Position

17: doThis name=Face basedOn ballLocation

18: end

1: begin lesson

2: start GetOpen hasId 8888

3: repeat

// Find your closest opponent.

4: doThis name=GetCurrentPosition

expect originalPosition

5: name=originalPosition isa type=Position

6: doThis name=FindClosestOpponent

basedOn=originalPosition

expect=opponentLocation

// Dash away from them

7: name=opponentLocation isa type=Position

8: doThis name=FaceAwayFrom

basedOn opponentLocation

9: doThis name=Dash expect=currentPosition

10: name=currentPosition isa type=Position

instructions that tellme can process10
Instructions that TellMe Can Process

// Stop and move back to your previous position

(e.g. cut back).

11: doThis name=MoveTowards

12: basedOnoriginalPosition expect=currentPosition

// If you are not open, do this again.

13: until

14: name=Open basedOncurrentPosition

// Once your open, find the ball and face it.

15: doThis name=FindTheBall expect=ballLocation

16: name=ballLocationisa type=Position

17: doThis name=Face basedOnballLocation

18: end

1: begin lesson

2: start GetOpenhasId 8888

3: repeat

// Find your closest opponent.

4: doThis name=GetCurrentPosition

expect originalPosition

5: name=originalPositionisa type=Position

6: doThis name=FindClosestOpponent

basedOn=originalPosition

expect=opponentLocation

// Dash away from them

7: name=opponentLocationisa type=Position

8: doThis name=FaceAwayFrom

basedOnopponentLocation

9: doThis name=Dash expect=currentPosition

10: name=currentPositionisa type=Position

applying the framework
Applying the Framework
  • “A Scientific Workflow Construction Command Line” [Groth & Gil IUI-09]
  • Real natural language descriptions of procedures:
    • Protocols in GenePattern [Reich et al 08]
    • Workflows in MyExperiment [DeRoure et al 09]
  • Example used:

“This workflow performs data cleansing on genes,

clusters the results,

and then displays a heatmap.”

conclusion
Conclusion
  • TellMe provides a framework for addressing learning from instruction given by humans
  • The approach can be applied to different domains
  • Future work includes:
    • More and better heuristics
    • Dealing with more sophisticated language constructs
    • Towards natural language input