Programming by Example - PowerPoint PPT Presentation

Programming by example l.jpg
1 / 72

  • Updated On :
  • Presentation posted in: General

Programming by Example. Henry Lieberman Media Laboratory Massachusetts Institute of Technology Cambridge, MA, USA PBE Web site: /PBE/. Your Wish is My Command: Programming by Example.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Programming by Example

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Programming by example l.jpg

Programming by Example

  • Henry Lieberman

  • Media Laboratory

  • Massachusetts Institute of Technology

  • Cambridge, MA, USA



  • PBE Web site: /PBE/

Your wish is my command programming by example l.jpg

Your Wish is My Command:Programming by Example

  • Morgan Kaufmann, 2001

Programming should be like teaching l.jpg

Programming should be like teaching

  • User plays the role of a teacher

  • Computer plays the role of a student

Programming by example4 l.jpg

Programming by Example

  • (Also called "Programming by Demonstration")

  • Agent "watches what you do" in an interactive interface

  • Records sequence of operations, data

  • Generalizes program so that you can use an analogous procedure in new examples

  • "Macros on Steroids"

Programming in the user interface l.jpg

Programming in the User Interface

  • The same interface used by end-users can be used for programming

  • Empowers end-users to do programming

  • No switching between user and programmer interfaces

  • Empowers "visual thinkers"

Pbe and the intelligent agent movement l.jpg

PBE and the "Intelligent Agent" movement

  • There's been a recent movement toward "intelligent agents" -- proactive, assistive, (possibly anthropomorphic) interfaces

  • PBE can be used to make agents instructible

  • Agent personalities:

  • Butler/Servant/Secretary

  • Teacher/Student, Master/Apprentice

Issues in programming by example l.jpg

Issues in Programming by Example

  • The Generalization Problem (= Data Description)

  • How do I describe actions and data?

  • How similar do the actions/data have to be next time?

  • Feedback

  • How does the system show the user what it's learned?

  • Advice

  • How can the user influence or modify what the system learns?

Issues in programming by example8 l.jpg

Issues in Programming by Example

  • How should we record the user's actions?

  • At what granularity should things be recorded?

  • Can the user interact with the agent during recording? If so, how?

  • Do we represent the agent directly in the interface?

  • Do we anthropomorphize the agent?

  • Can users give advice during recording?

  • Turning recording on and off

Issues in programming by example9 l.jpg

Issues in Programming by Example

  • How should the recorded program be represented?

  • Should the user see the recorded program?

  • What should it look like?

  • Should the user interact directly with the recorded program? If so, how?

  • How to represent examples?

  • How to represent generalizations?

Issues in programming by example10 l.jpg

Issues in Programming by Example

  • How should the user edit an already-recorded program?

  • How do we keep the representations consistent during editing?

  • Can the user edit previous generalizations?

  • How can users debug programs?

  • Can users step/trace/break programs?

Machine learning in programming by example l.jpg

Machine learning in Programming by Example

  • Lots of work in machine learning

  • Careful: Watch for hidden assumptions

  • Few examples, learns fast, explainable

    • Explanation-based Generalization

    • Case Based reasoning

  • Lots of examples, slow but more robust

    • Neural networks, PAC learning, data-mining

Users of the world unite l.jpg

Users of the world, unite!

  • You have nothing to lose but your error messages!

  • Almost "political" message of empowerment of users

  • Users should not be at the mercy of software providers

  • User procedures and data should not be held hostage by software companies

Poor man s programming by example l.jpg

"Poor man's"Programming by Example

  • Macro recorders in applications, AppleScript

  • Emacs keyboard macros

  • Interface builders / Scripting languages

  • Quickeys / OneClick …

Hooking pbe systems to conventional applications l.jpg

Hooking PBE systems to conventional applications

  • PBE systems would like to use conventional interfaces as tools rather than re-implement apps

  • Conventional interfaces aren't set up to be operated by an external program

  • Issues: Access to application data, sharing the interface, parallelism

  • Article: Lieberman, IUI 98 "Attaching Interface Agents to Conventional Applications"

Watch what i do l.jpg

Watch What I Do

  • Allen Cypher, ed. MIT Press 1993

A small matter of programming l.jpg

A Small Matter of Programming

  • Bonnie Nardi, MIT Press, 1994.

  • End-User Programming

  • Scripting languages

  • Application-specific languages

  • "Gardeners"

Eager allen cypher l.jpg

Eager - Allen Cypher

  • Programming by example for Hypercard

  • Notices patterns in user actions

  • Anticipation highlighting of predicted actions

  • Anthropomorphization of learning agent

  • Example: Collect subject lines from messages

Eager l.jpg


Mondrian an instructible graphical editor l.jpg

Mondrian: An Instructible Graphical Editor

  • Mondrian =

    • An object-oriented graphical editor +

    • An agent that records user interface actions +

    • Learning and generalization heuristics

Mondrian an instructible graphical editor20 l.jpg

Mondrian: An Instructible Graphical Editor

Wizard of oz experiment l.jpg

Wizard of Oz Experiment

  • Human simulates agent

  • What kinds of communication between human and agent are plausible?

  • What kind of knowledge does the agent need to know to perform the task?

Wizard of oz experiment task l.jpg

Wizard of Oz experiment task

  • Reformat a Bibliography

    • Lieberman, H., The One True Path to Intelligent Agents, International Conference on Buzzwords and Hype, Tahiti, 1996.

    • [Lieberman 96] The One True Path to Intelligent Agents, Int. Conf. on Buzzwords and Hype, Tahiti, (1996).

    • Oz, Wizard Of, Pay No Attention to the Man Behind the Curtain, Journal of Smoke and Mirrors, August 1942, Emerald City, Oz.

Wizard of oz results l.jpg

Wizard of Oz results

  • Users can make mistakes

  • Information can be incomplete

  • Agents can ask the user if they encounter an error situation

  • Agent could ask the user to supply missing information

  • Agent could ask the user to edit the example

  • Skip steps that don't work

Wizard of oz experiment method l.jpg

Wizard of Oz experiment method

  • User performs task once

    • “Thinking aloud”: goals and methods

  • Agent (secretly) constructs description of actions

    • In terms of simple “implementable” operations

  • Agent offers help

  • Agent performs help according to action description

  • Agent gives user feedback

  • Agent accepts advice from user

Survey of your wish systems l.jpg

Survey of "Your Wish" systems

  • Astonishingly wide variety of application domains:

  • Kids, Games, CAD, GIS, Text editing, Graphical editing, Web browsing, Animation, Databases, etc.

  • Illustrates variety of design decisions in PBE systems

  • Different reps for programs, agents, etc.

  • How much inference, generalization, initiative?

Stagecast creator smith cypher l.jpg

Stagecast Creator - Smith, Cypher

  • Programming for kids, grid game world

  • If you see this pattern of squares, replace it with that pattern of squares

  • Viz rep of rules, editor/stepper

  • Generalization on object types, introduce variables

  • Commercially available, success with thousands of users

A stagecast creator rule l.jpg

A Stagecast Creator Rule

A stagecast creator game l.jpg

A Stagecast Creator Game

Toontalk ken kahn l.jpg

Toontalk - Ken Kahn

  • Programming language is a video game

  • Most radical programming model

  • Concurrent logic programming

  • Abstraction by removing detail

  • Commercially available, success with kids in schools

Toontalk s visual language l.jpg

Toontalk's Visual Language

Generalizing by removing detail l.jpg

Generalizing by Removing Detail

Toontalk sort program l.jpg

Toontalk Sort program

How much intelligence brad myers rich mcdaniel l.jpg

How much intelligence?- Brad Myers, Rich McDaniel

  • CMU group one of most active and consistent in PBE

  • Multitude of systems, "gems" of PBE

  • Peridot, Lapidary, Jade, Gilt, Tourmaline, C32, Pursuit, Gold, Marquise, Katie, Turquoise, Topaz, Gamut

  • Experience with range of conservative to radical PBE systems

  • "Spectrum of Instructibility"

  • Tradeoff between conveience and control

Pursuit s visual language l.jpg

Pursuit's Visual Language

Selecting generalizations in marquise l.jpg

Selecting Generalizations in Marquise

Internet scrapbook sugiura l.jpg

Internet Scrapbook - Sugiura

  • Everybody's favorite application area - the Web!

  • Automates Browse & Cut & Paste applications

  • How do you describe a Web page selection?

  • Partial matching, Web design heuristics

  • Commercially available - preloaded on NEC PCs

Internet scrapbook l.jpg

Internet Scrapbook

Trias bauer meyer dengler paul l.jpg

TriAS - Bauer, Meyer, Dengler, Paul

  • Automating Browse & Copy & Paste, like Scrapbook

  • Collaborative, mixed-initiative dialog

  • At any time, agent or user can propose selections, generalizations

  • What Next? Can I Stop?

  • Integrates wrapper description language HyQL, InfoBeans dynamic Web sources

Collaborative wrapper definition l.jpg

Collaborative Wrapper Definition

Pbe for gis csprl traynor williams l.jpg

PBE for GIS: CSprl - Traynor, Williams

  • Study of how end users use Geographical Information Systems

  • "Gardeners", "Surrogate Users"

  • Comic-strip PBE language

A geographical info systems pbe program l.jpg

A Geographical Info Systems PBE Program

Visual language for geographic information systems l.jpg

Visual Language for Geographic Information Systems

Pbe for computer aided design girard pierra l.jpg

PBE for Computer Aided Design -Girard, Pierra

  • True "Industrial Strength" PBE, probably the champ

  • CAD users are sophisticated in application domain, willing to specify a lot

  • Provides complete editing and debugging

  • Key issues: Naming, Ambiguity in spec/solutions, general control structures: conditionals, loops

Example based programming in computer aided design l.jpg

Example-Based Programming in Computer-Aided Design

Demonstrating hidden features mcdaniel myers l.jpg

Demonstrating "Hidden Features" - McDaniel, Myers

  • PBE for animated computer games

  • WYSIWYG - "What you get is all you have"

  • Demonstrate "hidden features" by reifying them

  • Guides, e.g. Arrows for directions

  • Sophisticated heuristics, lots of inference

  • Stop This!

  • Do Something!

Demonstrating hidden features l.jpg

Demonstrating Hidden Features

Report generation masuishi l.jpg

Report Generation - Masuishi

  • Mundane, but hey, could save a lot of time

  • Format generalization

  • Application-specific iteration

  • Distributed to wide user community

Report generation by example l.jpg

Report Generation by Example

Dynamic macro masui l.jpg

Dynamic Macro - Masui

  • Notices loops in text editing commands

  • Simple, automatic: No start/stop recording

  • Wide community of users

  • Also: POBox predictive input for PDAs, phones, etc.

Dynamic macro l.jpg

Dynamic Macro

Smartedit lau domingos wolfman weld l.jpg

SmartEdit - Lau, Domingos, Wolfman, Weld

  • Automates repetitive text-editing procedures

  • Predictive, anticipatory, loop detection

  • Uses version-space learning

Smartedit version space l.jpg

SmartEdit Version Space

Smartedit screens l.jpg

SmartEdit Screens

Grammex grammars by example lieberman nardi wright l.jpg

Grammex: "Grammars by Example"- Lieberman, Nardi, Wright

  • Pick out "needles" of structured information in "haystack" of unstructured information

  • First direct-manipulation user interface to powerful grammar and parsing technology

  • Show example strings, explain components recursively

  • Iterative: System parses, you explain

Defining recursive grammars l.jpg

Defining Recursive Grammars

Visual representation of regular expressions blackwell l.jpg

Visual representation of regular expressions - Blackwell

  • SWYN: "See what you need"

  • Representation of positive and negative examples

  • Anticipation of selections updated dynamically

  • Representation of grammars as Finite-State machines

  • User study of understandability of representations

Visual representation of regular expressions l.jpg

Visual representation of Regular Expressions

Ape pbe for programming environments ruvini dony l.jpg

APE: PBE for Programming Environments - Ruvini, Dony

  • Combining predictive interfaces with PBE

  • Conventional programming environment: Smalltalk

  • Automates coding, debugging procedures

  • What-set, When-set use different learning algorithms

  • C4.5, new algorithm IDHYS can learn non-consecutive patterns

Repetitive code in protocols l.jpg

Repetitive code in protocols

Familiar pbe across applications paynter l.jpg

Familiar - PBE across applications - Paynter

  • PBE for Applescript

  • Application-independent, no mods to applications, uses commercial OS scripting language

  • Limited by shortcomings of Applescript and application implementations

Familiar arranging files l.jpg

Familiar arranging files

Tatlin gaxiola lieberman l.jpg

Tatlin - Gaxiola, Lieberman

  • Examinability: Some applications will let you look at their data even if you can't record them or use API

  • PBE by comparing states and inducing transformations rather than recording and generalizing actions

Tatlin pbe via examinability l.jpg

Tatlin - PBE via Examinability

Demonstrating what vs when wolber myers l.jpg

Demonstrating What vs. When- Wolber, Myers

  • Saying "when" you want something to happen

  • Stimulus-Response PBE

  • Pavlov: Animation scripting system like Macromind Director

Pavlov driving game l.jpg

Pavlov driving game

Programming by analogous examples repenning perrone l.jpg

Programming by Analogous Examples - Repenning, Perrone

  • Agentsheets - Grid world like Stagecast Creator

  • If-then rules by example

  • Make generalizations by analogy to generalizations of previous examples

Programming by analogous examples l.jpg

Programming by Analogous Examples

Visual generalization st amant zettlemoyer lieberman potter l.jpg

Visual Generalization - St. Amant, Zettlemoyer, Lieberman, Potter

  • Actually parse the screen!

  • Simulate human user's visual system

  • Finesses data access problem, hooking to applications

  • Fragile - if viz rep changes, you're screwed

  • Permits unique visual generalizations not possible by accessing data

Triggers just the pixels ma am l.jpg

Triggers - "Just the pixels, ma'am"

Solitaire by parsing the screen l.jpg

Solitaire by "Parsing the screen"

Short order programming exercise l.jpg

“Short order programming” exercise

  • Task: Add similarity-based learning to an action-based learning system

  • Ingredients: Graphical editor, action recording, relation inference

Things to think about l.jpg

Things to think about

  • How could we integrate the paradigms of Mondrian and Eager?

  • What sort of information do we need to get from the graphical editor?

  • What should the learning algorithm learn?

  • How should the result of learning affect the interface?

  • Login