programming by example l.
Skip this Video
Download Presentation
Programming by Example

Loading in 2 Seconds...

play fullscreen
1 / 72

Programming by Example - PowerPoint PPT Presentation

  • Uploaded on

Programming by Example. Henry Lieberman Media Laboratory Massachusetts Institute of Technology Cambridge, MA, USA PBE Web site: /PBE/. Your Wish is My Command: Programming by Example.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Programming by Example' - annot

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
programming by example
Programming by Example
  • Henry Lieberman
  • Media Laboratory
  • Massachusetts Institute of Technology
  • Cambridge, MA, USA
  • PBE Web site: /PBE/
programming should be like teaching
Programming should be like teaching
  • User plays the role of a teacher
  • Computer plays the role of a student
programming by example4
Programming by Example
  • (Also called "Programming by Demonstration")
  • Agent "watches what you do" in an interactive interface
  • Records sequence of operations, data
  • Generalizes program so that you can use an analogous procedure in new examples
  • "Macros on Steroids"
programming in the user interface
Programming in the User Interface
  • The same interface used by end-users can be used for programming
  • Empowers end-users to do programming
  • No switching between user and programmer interfaces
  • Empowers "visual thinkers"
pbe and the intelligent agent movement
PBE and the "Intelligent Agent" movement
  • There's been a recent movement toward "intelligent agents" -- proactive, assistive, (possibly anthropomorphic) interfaces
  • PBE can be used to make agents instructible
  • Agent personalities:
  • Butler/Servant/Secretary
  • Teacher/Student, Master/Apprentice
issues in programming by example
Issues in Programming by Example
  • The Generalization Problem (= Data Description)
  • How do I describe actions and data?
  • How similar do the actions/data have to be next time?
  • Feedback
  • How does the system show the user what it's learned?
  • Advice
  • How can the user influence or modify what the system learns?
issues in programming by example8
Issues in Programming by Example
  • How should we record the user's actions?
  • At what granularity should things be recorded?
  • Can the user interact with the agent during recording? If so, how?
  • Do we represent the agent directly in the interface?
  • Do we anthropomorphize the agent?
  • Can users give advice during recording?
  • Turning recording on and off
issues in programming by example9
Issues in Programming by Example
  • How should the recorded program be represented?
  • Should the user see the recorded program?
  • What should it look like?
  • Should the user interact directly with the recorded program? If so, how?
  • How to represent examples?
  • How to represent generalizations?
issues in programming by example10
Issues in Programming by Example
  • How should the user edit an already-recorded program?
  • How do we keep the representations consistent during editing?
  • Can the user edit previous generalizations?
  • How can users debug programs?
  • Can users step/trace/break programs?
machine learning in programming by example
Machine learning in Programming by Example
  • Lots of work in machine learning
  • Careful: Watch for hidden assumptions
  • Few examples, learns fast, explainable
    • Explanation-based Generalization
    • Case Based reasoning
  • Lots of examples, slow but more robust
    • Neural networks, PAC learning, data-mining
users of the world unite
Users of the world, unite!
  • You have nothing to lose but your error messages!
  • Almost "political" message of empowerment of users
  • Users should not be at the mercy of software providers
  • User procedures and data should not be held hostage by software companies
poor man s programming by example
"Poor man's"Programming by Example
  • Macro recorders in applications, AppleScript
  • Emacs keyboard macros
  • Interface builders / Scripting languages
  • Quickeys / OneClick …
hooking pbe systems to conventional applications
Hooking PBE systems to conventional applications
  • PBE systems would like to use conventional interfaces as tools rather than re-implement apps
  • Conventional interfaces aren't set up to be operated by an external program
  • Issues: Access to application data, sharing the interface, parallelism
  • Article: Lieberman, IUI 98 "Attaching Interface Agents to Conventional Applications"
watch what i do
Watch What I Do
  • Allen Cypher, ed. MIT Press 1993
a small matter of programming
A Small Matter of Programming
  • Bonnie Nardi, MIT Press, 1994.
  • End-User Programming
  • Scripting languages
  • Application-specific languages
  • "Gardeners"
eager allen cypher
Eager - Allen Cypher
  • Programming by example for Hypercard
  • Notices patterns in user actions
  • Anticipation highlighting of predicted actions
  • Anthropomorphization of learning agent
  • Example: Collect subject lines from messages
mondrian an instructible graphical editor
Mondrian: An Instructible Graphical Editor
  • Mondrian =
    • An object-oriented graphical editor +
    • An agent that records user interface actions +
    • Learning and generalization heuristics
wizard of oz experiment
Wizard of Oz Experiment
  • Human simulates agent
  • What kinds of communication between human and agent are plausible?
  • What kind of knowledge does the agent need to know to perform the task?
wizard of oz experiment task
Wizard of Oz experiment task
  • Reformat a Bibliography
    • Lieberman, H., The One True Path to Intelligent Agents, International Conference on Buzzwords and Hype, Tahiti, 1996.
    • [Lieberman 96] The One True Path to Intelligent Agents, Int. Conf. on Buzzwords and Hype, Tahiti, (1996).
    • Oz, Wizard Of, Pay No Attention to the Man Behind the Curtain, Journal of Smoke and Mirrors, August 1942, Emerald City, Oz.
wizard of oz results
Wizard of Oz results
  • Users can make mistakes
  • Information can be incomplete
  • Agents can ask the user if they encounter an error situation
  • Agent could ask the user to supply missing information
  • Agent could ask the user to edit the example
  • Skip steps that don't work
wizard of oz experiment method
Wizard of Oz experiment method
  • User performs task once
    • “Thinking aloud”: goals and methods
  • Agent (secretly) constructs description of actions
    • In terms of simple “implementable” operations
  • Agent offers help
  • Agent performs help according to action description
  • Agent gives user feedback
  • Agent accepts advice from user
survey of your wish systems
Survey of "Your Wish" systems
  • Astonishingly wide variety of application domains:
  • Kids, Games, CAD, GIS, Text editing, Graphical editing, Web browsing, Animation, Databases, etc.
  • Illustrates variety of design decisions in PBE systems
  • Different reps for programs, agents, etc.
  • How much inference, generalization, initiative?
stagecast creator smith cypher
Stagecast Creator - Smith, Cypher
  • Programming for kids, grid game world
  • If you see this pattern of squares, replace it with that pattern of squares
  • Viz rep of rules, editor/stepper
  • Generalization on object types, introduce variables
  • Commercially available, success with thousands of users
toontalk ken kahn
Toontalk - Ken Kahn
  • Programming language is a video game
  • Most radical programming model
  • Concurrent logic programming
  • Abstraction by removing detail
  • Commercially available, success with kids in schools
how much intelligence brad myers rich mcdaniel
How much intelligence?- Brad Myers, Rich McDaniel
  • CMU group one of most active and consistent in PBE
  • Multitude of systems, "gems" of PBE
  • Peridot, Lapidary, Jade, Gilt, Tourmaline, C32, Pursuit, Gold, Marquise, Katie, Turquoise, Topaz, Gamut
  • Experience with range of conservative to radical PBE systems
  • "Spectrum of Instructibility"
  • Tradeoff between conveience and control
internet scrapbook sugiura
Internet Scrapbook - Sugiura
  • Everybody's favorite application area - the Web!
  • Automates Browse & Cut & Paste applications
  • How do you describe a Web page selection?
  • Partial matching, Web design heuristics
  • Commercially available - preloaded on NEC PCs
trias bauer meyer dengler paul
TriAS - Bauer, Meyer, Dengler, Paul
  • Automating Browse & Copy & Paste, like Scrapbook
  • Collaborative, mixed-initiative dialog
  • At any time, agent or user can propose selections, generalizations
  • What Next? Can I Stop?
  • Integrates wrapper description language HyQL, InfoBeans dynamic Web sources
pbe for gis csprl traynor williams
PBE for GIS: CSprl - Traynor, Williams
  • Study of how end users use Geographical Information Systems
  • "Gardeners", "Surrogate Users"
  • Comic-strip PBE language
pbe for computer aided design girard pierra
PBE for Computer Aided Design -Girard, Pierra
  • True "Industrial Strength" PBE, probably the champ
  • CAD users are sophisticated in application domain, willing to specify a lot
  • Provides complete editing and debugging
  • Key issues: Naming, Ambiguity in spec/solutions, general control structures: conditionals, loops
demonstrating hidden features mcdaniel myers
Demonstrating "Hidden Features" - McDaniel, Myers
  • PBE for animated computer games
  • WYSIWYG - "What you get is all you have"
  • Demonstrate "hidden features" by reifying them
  • Guides, e.g. Arrows for directions
  • Sophisticated heuristics, lots of inference
  • Stop This!
  • Do Something!
report generation masuishi
Report Generation - Masuishi
  • Mundane, but hey, could save a lot of time
  • Format generalization
  • Application-specific iteration
  • Distributed to wide user community
dynamic macro masui
Dynamic Macro - Masui
  • Notices loops in text editing commands
  • Simple, automatic: No start/stop recording
  • Wide community of users
  • Also: POBox predictive input for PDAs, phones, etc.
smartedit lau domingos wolfman weld
SmartEdit - Lau, Domingos, Wolfman, Weld
  • Automates repetitive text-editing procedures
  • Predictive, anticipatory, loop detection
  • Uses version-space learning
grammex grammars by example lieberman nardi wright
Grammex: "Grammars by Example"- Lieberman, Nardi, Wright
  • Pick out "needles" of structured information in "haystack" of unstructured information
  • First direct-manipulation user interface to powerful grammar and parsing technology
  • Show example strings, explain components recursively
  • Iterative: System parses, you explain
visual representation of regular expressions blackwell
Visual representation of regular expressions - Blackwell
  • SWYN: "See what you need"
  • Representation of positive and negative examples
  • Anticipation of selections updated dynamically
  • Representation of grammars as Finite-State machines
  • User study of understandability of representations
ape pbe for programming environments ruvini dony
APE: PBE for Programming Environments - Ruvini, Dony
  • Combining predictive interfaces with PBE
  • Conventional programming environment: Smalltalk
  • Automates coding, debugging procedures
  • What-set, When-set use different learning algorithms
  • C4.5, new algorithm IDHYS can learn non-consecutive patterns
familiar pbe across applications paynter
Familiar - PBE across applications - Paynter
  • PBE for Applescript
  • Application-independent, no mods to applications, uses commercial OS scripting language
  • Limited by shortcomings of Applescript and application implementations
tatlin gaxiola lieberman
Tatlin - Gaxiola, Lieberman
  • Examinability: Some applications will let you look at their data even if you can't record them or use API
  • PBE by comparing states and inducing transformations rather than recording and generalizing actions
demonstrating what vs when wolber myers
Demonstrating What vs. When- Wolber, Myers
  • Saying "when" you want something to happen
  • Stimulus-Response PBE
  • Pavlov: Animation scripting system like Macromind Director
programming by analogous examples repenning perrone
Programming by Analogous Examples - Repenning, Perrone
  • Agentsheets - Grid world like Stagecast Creator
  • If-then rules by example
  • Make generalizations by analogy to generalizations of previous examples
visual generalization st amant zettlemoyer lieberman potter
Visual Generalization - St. Amant, Zettlemoyer, Lieberman, Potter
  • Actually parse the screen!
  • Simulate human user's visual system
  • Finesses data access problem, hooking to applications
  • Fragile - if viz rep changes, you're screwed
  • Permits unique visual generalizations not possible by accessing data
short order programming exercise
“Short order programming” exercise
  • Task: Add similarity-based learning to an action-based learning system
  • Ingredients: Graphical editor, action recording, relation inference
things to think about
Things to think about
  • How could we integrate the paradigms of Mondrian and Eager?
  • What sort of information do we need to get from the graphical editor?
  • What should the learning algorithm learn?
  • How should the result of learning affect the interface?