Programming by example l.jpg
Sponsored Links
This presentation is the property of its rightful owner.
1 / 72

Programming by Example PowerPoint PPT Presentation


  • 82 Views
  • Uploaded on
  • Presentation posted in: General

Programming by Example. Henry Lieberman Media Laboratory Massachusetts Institute of Technology Cambridge, MA, USA [email protected] http://www.media.mit.edu/~lieber PBE Web site: http://www.media.mit.edu/~lieber /PBE/. Your Wish is My Command: Programming by Example.

Download Presentation

Programming by Example

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Programming by Example

  • Henry Lieberman

  • Media Laboratory

  • Massachusetts Institute of Technology

  • Cambridge, MA, USA

  • [email protected]

  • http://www.media.mit.edu/~lieber

  • PBE Web site: http://www.media.mit.edu/~lieber /PBE/


Your Wish is My Command:Programming by Example

  • Morgan Kaufmann, 2001


Programming should be like teaching

  • User plays the role of a teacher

  • Computer plays the role of a student


Programming by Example

  • (Also called "Programming by Demonstration")

  • Agent "watches what you do" in an interactive interface

  • Records sequence of operations, data

  • Generalizes program so that you can use an analogous procedure in new examples

  • "Macros on Steroids"


Programming in the User Interface

  • The same interface used by end-users can be used for programming

  • Empowers end-users to do programming

  • No switching between user and programmer interfaces

  • Empowers "visual thinkers"


PBE and the "Intelligent Agent" movement

  • There's been a recent movement toward "intelligent agents" -- proactive, assistive, (possibly anthropomorphic) interfaces

  • PBE can be used to make agents instructible

  • Agent personalities:

  • Butler/Servant/Secretary

  • Teacher/Student, Master/Apprentice


Issues in Programming by Example

  • The Generalization Problem (= Data Description)

  • How do I describe actions and data?

  • How similar do the actions/data have to be next time?

  • Feedback

  • How does the system show the user what it's learned?

  • Advice

  • How can the user influence or modify what the system learns?


Issues in Programming by Example

  • How should we record the user's actions?

  • At what granularity should things be recorded?

  • Can the user interact with the agent during recording? If so, how?

  • Do we represent the agent directly in the interface?

  • Do we anthropomorphize the agent?

  • Can users give advice during recording?

  • Turning recording on and off


Issues in Programming by Example

  • How should the recorded program be represented?

  • Should the user see the recorded program?

  • What should it look like?

  • Should the user interact directly with the recorded program? If so, how?

  • How to represent examples?

  • How to represent generalizations?


Issues in Programming by Example

  • How should the user edit an already-recorded program?

  • How do we keep the representations consistent during editing?

  • Can the user edit previous generalizations?

  • How can users debug programs?

  • Can users step/trace/break programs?


Machine learning in Programming by Example

  • Lots of work in machine learning

  • Careful: Watch for hidden assumptions

  • Few examples, learns fast, explainable

    • Explanation-based Generalization

    • Case Based reasoning

  • Lots of examples, slow but more robust

    • Neural networks, PAC learning, data-mining


Users of the world, unite!

  • You have nothing to lose but your error messages!

  • Almost "political" message of empowerment of users

  • Users should not be at the mercy of software providers

  • User procedures and data should not be held hostage by software companies


"Poor man's"Programming by Example

  • Macro recorders in applications, AppleScript

  • Emacs keyboard macros

  • Interface builders / Scripting languages

  • Quickeys / OneClick …


Hooking PBE systems to conventional applications

  • PBE systems would like to use conventional interfaces as tools rather than re-implement apps

  • Conventional interfaces aren't set up to be operated by an external program

  • Issues: Access to application data, sharing the interface, parallelism

  • Article: Lieberman, IUI 98 "Attaching Interface Agents to Conventional Applications"


Watch What I Do

  • Allen Cypher, ed. MIT Press 1993


A Small Matter of Programming

  • Bonnie Nardi, MIT Press, 1994.

  • End-User Programming

  • Scripting languages

  • Application-specific languages

  • "Gardeners"


Eager - Allen Cypher

  • Programming by example for Hypercard

  • Notices patterns in user actions

  • Anticipation highlighting of predicted actions

  • Anthropomorphization of learning agent

  • Example: Collect subject lines from messages


Eager


Mondrian: An Instructible Graphical Editor

  • Mondrian =

    • An object-oriented graphical editor +

    • An agent that records user interface actions +

    • Learning and generalization heuristics


Mondrian: An Instructible Graphical Editor


Wizard of Oz Experiment

  • Human simulates agent

  • What kinds of communication between human and agent are plausible?

  • What kind of knowledge does the agent need to know to perform the task?


Wizard of Oz experiment task

  • Reformat a Bibliography

    • Lieberman, H., The One True Path to Intelligent Agents, International Conference on Buzzwords and Hype, Tahiti, 1996.

    • [Lieberman 96] The One True Path to Intelligent Agents, Int. Conf. on Buzzwords and Hype, Tahiti, (1996).

    • Oz, Wizard Of, Pay No Attention to the Man Behind the Curtain, Journal of Smoke and Mirrors, August 1942, Emerald City, Oz.


Wizard of Oz results

  • Users can make mistakes

  • Information can be incomplete

  • Agents can ask the user if they encounter an error situation

  • Agent could ask the user to supply missing information

  • Agent could ask the user to edit the example

  • Skip steps that don't work


Wizard of Oz experiment method

  • User performs task once

    • “Thinking aloud”: goals and methods

  • Agent (secretly) constructs description of actions

    • In terms of simple “implementable” operations

  • Agent offers help

  • Agent performs help according to action description

  • Agent gives user feedback

  • Agent accepts advice from user


Survey of "Your Wish" systems

  • Astonishingly wide variety of application domains:

  • Kids, Games, CAD, GIS, Text editing, Graphical editing, Web browsing, Animation, Databases, etc.

  • Illustrates variety of design decisions in PBE systems

  • Different reps for programs, agents, etc.

  • How much inference, generalization, initiative?


Stagecast Creator - Smith, Cypher

  • Programming for kids, grid game world

  • If you see this pattern of squares, replace it with that pattern of squares

  • Viz rep of rules, editor/stepper

  • Generalization on object types, introduce variables

  • Commercially available, success with thousands of users


A Stagecast Creator Rule


A Stagecast Creator Game


Toontalk - Ken Kahn

  • Programming language is a video game

  • Most radical programming model

  • Concurrent logic programming

  • Abstraction by removing detail

  • Commercially available, success with kids in schools


Toontalk's Visual Language


Generalizing by Removing Detail


Toontalk Sort program


How much intelligence?- Brad Myers, Rich McDaniel

  • CMU group one of most active and consistent in PBE

  • Multitude of systems, "gems" of PBE

  • Peridot, Lapidary, Jade, Gilt, Tourmaline, C32, Pursuit, Gold, Marquise, Katie, Turquoise, Topaz, Gamut

  • Experience with range of conservative to radical PBE systems

  • "Spectrum of Instructibility"

  • Tradeoff between conveience and control


Pursuit's Visual Language


Selecting Generalizations in Marquise


Internet Scrapbook - Sugiura

  • Everybody's favorite application area - the Web!

  • Automates Browse & Cut & Paste applications

  • How do you describe a Web page selection?

  • Partial matching, Web design heuristics

  • Commercially available - preloaded on NEC PCs


Internet Scrapbook


TriAS - Bauer, Meyer, Dengler, Paul

  • Automating Browse & Copy & Paste, like Scrapbook

  • Collaborative, mixed-initiative dialog

  • At any time, agent or user can propose selections, generalizations

  • What Next? Can I Stop?

  • Integrates wrapper description language HyQL, InfoBeans dynamic Web sources


Collaborative Wrapper Definition


PBE for GIS: CSprl - Traynor, Williams

  • Study of how end users use Geographical Information Systems

  • "Gardeners", "Surrogate Users"

  • Comic-strip PBE language


A Geographical Info Systems PBE Program


Visual Language for Geographic Information Systems


PBE for Computer Aided Design -Girard, Pierra

  • True "Industrial Strength" PBE, probably the champ

  • CAD users are sophisticated in application domain, willing to specify a lot

  • Provides complete editing and debugging

  • Key issues: Naming, Ambiguity in spec/solutions, general control structures: conditionals, loops


Example-Based Programming in Computer-Aided Design


Demonstrating "Hidden Features" - McDaniel, Myers

  • PBE for animated computer games

  • WYSIWYG - "What you get is all you have"

  • Demonstrate "hidden features" by reifying them

  • Guides, e.g. Arrows for directions

  • Sophisticated heuristics, lots of inference

  • Stop This!

  • Do Something!


Demonstrating Hidden Features


Report Generation - Masuishi

  • Mundane, but hey, could save a lot of time

  • Format generalization

  • Application-specific iteration

  • Distributed to wide user community


Report Generation by Example


Dynamic Macro - Masui

  • Notices loops in text editing commands

  • Simple, automatic: No start/stop recording

  • Wide community of users

  • Also: POBox predictive input for PDAs, phones, etc.


Dynamic Macro


SmartEdit - Lau, Domingos, Wolfman, Weld

  • Automates repetitive text-editing procedures

  • Predictive, anticipatory, loop detection

  • Uses version-space learning


SmartEdit Version Space


SmartEdit Screens


Grammex: "Grammars by Example"- Lieberman, Nardi, Wright

  • Pick out "needles" of structured information in "haystack" of unstructured information

  • First direct-manipulation user interface to powerful grammar and parsing technology

  • Show example strings, explain components recursively

  • Iterative: System parses, you explain


Defining Recursive Grammars


Visual representation of regular expressions - Blackwell

  • SWYN: "See what you need"

  • Representation of positive and negative examples

  • Anticipation of selections updated dynamically

  • Representation of grammars as Finite-State machines

  • User study of understandability of representations


Visual representation of Regular Expressions


APE: PBE for Programming Environments - Ruvini, Dony

  • Combining predictive interfaces with PBE

  • Conventional programming environment: Smalltalk

  • Automates coding, debugging procedures

  • What-set, When-set use different learning algorithms

  • C4.5, new algorithm IDHYS can learn non-consecutive patterns


Repetitive code in protocols


Familiar - PBE across applications - Paynter

  • PBE for Applescript

  • Application-independent, no mods to applications, uses commercial OS scripting language

  • Limited by shortcomings of Applescript and application implementations


Familiar arranging files


Tatlin - Gaxiola, Lieberman

  • Examinability: Some applications will let you look at their data even if you can't record them or use API

  • PBE by comparing states and inducing transformations rather than recording and generalizing actions


Tatlin - PBE via Examinability


Demonstrating What vs. When- Wolber, Myers

  • Saying "when" you want something to happen

  • Stimulus-Response PBE

  • Pavlov: Animation scripting system like Macromind Director


Pavlov driving game


Programming by Analogous Examples - Repenning, Perrone

  • Agentsheets - Grid world like Stagecast Creator

  • If-then rules by example

  • Make generalizations by analogy to generalizations of previous examples


Programming by Analogous Examples


Visual Generalization - St. Amant, Zettlemoyer, Lieberman, Potter

  • Actually parse the screen!

  • Simulate human user's visual system

  • Finesses data access problem, hooking to applications

  • Fragile - if viz rep changes, you're screwed

  • Permits unique visual generalizations not possible by accessing data


Triggers - "Just the pixels, ma'am"


Solitaire by "Parsing the screen"


“Short order programming” exercise

  • Task: Add similarity-based learning to an action-based learning system

  • Ingredients: Graphical editor, action recording, relation inference


Things to think about

  • How could we integrate the paradigms of Mondrian and Eager?

  • What sort of information do we need to get from the graphical editor?

  • What should the learning algorithm learn?

  • How should the result of learning affect the interface?


  • Login