Programming by example
1 / 72

Programming by Example - PowerPoint PPT Presentation

  • Updated On :

Programming by Example. Henry Lieberman Media Laboratory Massachusetts Institute of Technology Cambridge, MA, USA [email protected] PBE Web site: /PBE/. Your Wish is My Command: Programming by Example.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Programming by Example' - annot

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Programming by example l.jpg
Programming by Example

  • Henry Lieberman

  • Media Laboratory

  • Massachusetts Institute of Technology

  • Cambridge, MA, USA

  • [email protected]


  • PBE Web site: /PBE/

Your wish is my command programming by example l.jpg
Your Wish is My Command:Programming by Example

  • Morgan Kaufmann, 2001

Programming should be like teaching l.jpg
Programming should be like teaching

  • User plays the role of a teacher

  • Computer plays the role of a student

Programming by example4 l.jpg
Programming by Example

  • (Also called "Programming by Demonstration")

  • Agent "watches what you do" in an interactive interface

  • Records sequence of operations, data

  • Generalizes program so that you can use an analogous procedure in new examples

  • "Macros on Steroids"

Programming in the user interface l.jpg
Programming in the User Interface

  • The same interface used by end-users can be used for programming

  • Empowers end-users to do programming

  • No switching between user and programmer interfaces

  • Empowers "visual thinkers"

Pbe and the intelligent agent movement l.jpg
PBE and the "Intelligent Agent" movement

  • There's been a recent movement toward "intelligent agents" -- proactive, assistive, (possibly anthropomorphic) interfaces

  • PBE can be used to make agents instructible

  • Agent personalities:

  • Butler/Servant/Secretary

  • Teacher/Student, Master/Apprentice

Issues in programming by example l.jpg
Issues in Programming by Example

  • The Generalization Problem (= Data Description)

  • How do I describe actions and data?

  • How similar do the actions/data have to be next time?

  • Feedback

  • How does the system show the user what it's learned?

  • Advice

  • How can the user influence or modify what the system learns?

Issues in programming by example8 l.jpg
Issues in Programming by Example

  • How should we record the user's actions?

  • At what granularity should things be recorded?

  • Can the user interact with the agent during recording? If so, how?

  • Do we represent the agent directly in the interface?

  • Do we anthropomorphize the agent?

  • Can users give advice during recording?

  • Turning recording on and off

Issues in programming by example9 l.jpg
Issues in Programming by Example

  • How should the recorded program be represented?

  • Should the user see the recorded program?

  • What should it look like?

  • Should the user interact directly with the recorded program? If so, how?

  • How to represent examples?

  • How to represent generalizations?

Issues in programming by example10 l.jpg
Issues in Programming by Example

  • How should the user edit an already-recorded program?

  • How do we keep the representations consistent during editing?

  • Can the user edit previous generalizations?

  • How can users debug programs?

  • Can users step/trace/break programs?

Machine learning in programming by example l.jpg
Machine learning in Programming by Example

  • Lots of work in machine learning

  • Careful: Watch for hidden assumptions

  • Few examples, learns fast, explainable

    • Explanation-based Generalization

    • Case Based reasoning

  • Lots of examples, slow but more robust

    • Neural networks, PAC learning, data-mining

Users of the world unite l.jpg
Users of the world, unite!

  • You have nothing to lose but your error messages!

  • Almost "political" message of empowerment of users

  • Users should not be at the mercy of software providers

  • User procedures and data should not be held hostage by software companies

Poor man s programming by example l.jpg
"Poor man's"Programming by Example

  • Macro recorders in applications, AppleScript

  • Emacs keyboard macros

  • Interface builders / Scripting languages

  • Quickeys / OneClick …

Hooking pbe systems to conventional applications l.jpg
Hooking PBE systems to conventional applications

  • PBE systems would like to use conventional interfaces as tools rather than re-implement apps

  • Conventional interfaces aren't set up to be operated by an external program

  • Issues: Access to application data, sharing the interface, parallelism

  • Article: Lieberman, IUI 98 "Attaching Interface Agents to Conventional Applications"

Watch what i do l.jpg
Watch What I Do

  • Allen Cypher, ed. MIT Press 1993

A small matter of programming l.jpg
A Small Matter of Programming

  • Bonnie Nardi, MIT Press, 1994.

  • End-User Programming

  • Scripting languages

  • Application-specific languages

  • "Gardeners"

Eager allen cypher l.jpg
Eager - Allen Cypher

  • Programming by example for Hypercard

  • Notices patterns in user actions

  • Anticipation highlighting of predicted actions

  • Anthropomorphization of learning agent

  • Example: Collect subject lines from messages

Mondrian an instructible graphical editor l.jpg
Mondrian: An Instructible Graphical Editor

  • Mondrian =

    • An object-oriented graphical editor +

    • An agent that records user interface actions +

    • Learning and generalization heuristics

Mondrian an instructible graphical editor20 l.jpg
Mondrian: An Instructible Graphical Editor

Wizard of oz experiment l.jpg
Wizard of Oz Experiment

  • Human simulates agent

  • What kinds of communication between human and agent are plausible?

  • What kind of knowledge does the agent need to know to perform the task?

Wizard of oz experiment task l.jpg
Wizard of Oz experiment task

  • Reformat a Bibliography

    • Lieberman, H., The One True Path to Intelligent Agents, International Conference on Buzzwords and Hype, Tahiti, 1996.

    • [Lieberman 96] The One True Path to Intelligent Agents, Int. Conf. on Buzzwords and Hype, Tahiti, (1996).

    • Oz, Wizard Of, Pay No Attention to the Man Behind the Curtain, Journal of Smoke and Mirrors, August 1942, Emerald City, Oz.

Wizard of oz results l.jpg
Wizard of Oz results

  • Users can make mistakes

  • Information can be incomplete

  • Agents can ask the user if they encounter an error situation

  • Agent could ask the user to supply missing information

  • Agent could ask the user to edit the example

  • Skip steps that don't work

Wizard of oz experiment method l.jpg
Wizard of Oz experiment method

  • User performs task once

    • “Thinking aloud”: goals and methods

  • Agent (secretly) constructs description of actions

    • In terms of simple “implementable” operations

  • Agent offers help

  • Agent performs help according to action description

  • Agent gives user feedback

  • Agent accepts advice from user

Survey of your wish systems l.jpg
Survey of "Your Wish" systems

  • Astonishingly wide variety of application domains:

  • Kids, Games, CAD, GIS, Text editing, Graphical editing, Web browsing, Animation, Databases, etc.

  • Illustrates variety of design decisions in PBE systems

  • Different reps for programs, agents, etc.

  • How much inference, generalization, initiative?

Stagecast creator smith cypher l.jpg
Stagecast Creator - Smith, Cypher

  • Programming for kids, grid game world

  • If you see this pattern of squares, replace it with that pattern of squares

  • Viz rep of rules, editor/stepper

  • Generalization on object types, introduce variables

  • Commercially available, success with thousands of users

Toontalk ken kahn l.jpg
Toontalk - Ken Kahn

  • Programming language is a video game

  • Most radical programming model

  • Concurrent logic programming

  • Abstraction by removing detail

  • Commercially available, success with kids in schools

How much intelligence brad myers rich mcdaniel l.jpg
How much intelligence?- Brad Myers, Rich McDaniel

  • CMU group one of most active and consistent in PBE

  • Multitude of systems, "gems" of PBE

  • Peridot, Lapidary, Jade, Gilt, Tourmaline, C32, Pursuit, Gold, Marquise, Katie, Turquoise, Topaz, Gamut

  • Experience with range of conservative to radical PBE systems

  • "Spectrum of Instructibility"

  • Tradeoff between conveience and control

Internet scrapbook sugiura l.jpg
Internet Scrapbook - Sugiura

  • Everybody's favorite application area - the Web!

  • Automates Browse & Cut & Paste applications

  • How do you describe a Web page selection?

  • Partial matching, Web design heuristics

  • Commercially available - preloaded on NEC PCs

Trias bauer meyer dengler paul l.jpg
TriAS - Bauer, Meyer, Dengler, Paul

  • Automating Browse & Copy & Paste, like Scrapbook

  • Collaborative, mixed-initiative dialog

  • At any time, agent or user can propose selections, generalizations

  • What Next? Can I Stop?

  • Integrates wrapper description language HyQL, InfoBeans dynamic Web sources

Pbe for gis csprl traynor williams l.jpg
PBE for GIS: CSprl - Traynor, Williams

  • Study of how end users use Geographical Information Systems

  • "Gardeners", "Surrogate Users"

  • Comic-strip PBE language

Pbe for computer aided design girard pierra l.jpg
PBE for Computer Aided Design -Girard, Pierra

  • True "Industrial Strength" PBE, probably the champ

  • CAD users are sophisticated in application domain, willing to specify a lot

  • Provides complete editing and debugging

  • Key issues: Naming, Ambiguity in spec/solutions, general control structures: conditionals, loops

Demonstrating hidden features mcdaniel myers l.jpg
Demonstrating "Hidden Features" - McDaniel, Myers

  • PBE for animated computer games

  • WYSIWYG - "What you get is all you have"

  • Demonstrate "hidden features" by reifying them

  • Guides, e.g. Arrows for directions

  • Sophisticated heuristics, lots of inference

  • Stop This!

  • Do Something!

Report generation masuishi l.jpg
Report Generation - Masuishi

  • Mundane, but hey, could save a lot of time

  • Format generalization

  • Application-specific iteration

  • Distributed to wide user community

Dynamic macro masui l.jpg
Dynamic Macro - Masui

  • Notices loops in text editing commands

  • Simple, automatic: No start/stop recording

  • Wide community of users

  • Also: POBox predictive input for PDAs, phones, etc.

Smartedit lau domingos wolfman weld l.jpg
SmartEdit - Lau, Domingos, Wolfman, Weld

  • Automates repetitive text-editing procedures

  • Predictive, anticipatory, loop detection

  • Uses version-space learning

Grammex grammars by example lieberman nardi wright l.jpg
Grammex: "Grammars by Example"- Lieberman, Nardi, Wright

  • Pick out "needles" of structured information in "haystack" of unstructured information

  • First direct-manipulation user interface to powerful grammar and parsing technology

  • Show example strings, explain components recursively

  • Iterative: System parses, you explain

Visual representation of regular expressions blackwell l.jpg
Visual representation of regular expressions - Blackwell

  • SWYN: "See what you need"

  • Representation of positive and negative examples

  • Anticipation of selections updated dynamically

  • Representation of grammars as Finite-State machines

  • User study of understandability of representations

Visual representation of regular expressions l.jpg
Visual representation of Regular Expressions

Ape pbe for programming environments ruvini dony l.jpg
APE: PBE for Programming Environments - Ruvini, Dony

  • Combining predictive interfaces with PBE

  • Conventional programming environment: Smalltalk

  • Automates coding, debugging procedures

  • What-set, When-set use different learning algorithms

  • C4.5, new algorithm IDHYS can learn non-consecutive patterns

Familiar pbe across applications paynter l.jpg
Familiar - PBE across applications - Paynter

  • PBE for Applescript

  • Application-independent, no mods to applications, uses commercial OS scripting language

  • Limited by shortcomings of Applescript and application implementations

Tatlin gaxiola lieberman l.jpg
Tatlin - Gaxiola, Lieberman

  • Examinability: Some applications will let you look at their data even if you can't record them or use API

  • PBE by comparing states and inducing transformations rather than recording and generalizing actions

Demonstrating what vs when wolber myers l.jpg
Demonstrating What vs. When- Wolber, Myers

  • Saying "when" you want something to happen

  • Stimulus-Response PBE

  • Pavlov: Animation scripting system like Macromind Director

Programming by analogous examples repenning perrone l.jpg
Programming by Analogous Examples - Repenning, Perrone

  • Agentsheets - Grid world like Stagecast Creator

  • If-then rules by example

  • Make generalizations by analogy to generalizations of previous examples

Programming by analogous examples l.jpg
Programming by Analogous Examples

Visual generalization st amant zettlemoyer lieberman potter l.jpg
Visual Generalization - St. Amant, Zettlemoyer, Lieberman, Potter

  • Actually parse the screen!

  • Simulate human user's visual system

  • Finesses data access problem, hooking to applications

  • Fragile - if viz rep changes, you're screwed

  • Permits unique visual generalizations not possible by accessing data

Short order programming exercise l.jpg
“Short order programming” exercise Potter

  • Task: Add similarity-based learning to an action-based learning system

  • Ingredients: Graphical editor, action recording, relation inference

Things to think about l.jpg
Things to think about Potter

  • How could we integrate the paradigms of Mondrian and Eager?

  • What sort of information do we need to get from the graphical editor?

  • What should the learning algorithm learn?

  • How should the result of learning affect the interface?