Symmetric multi modal intelligent interactive development environment smiide seedling
1 / 20

Symmetric Multi-modal Intelligent Interactive Development Environment (SMIIDE) Seedling - PowerPoint PPT Presentation

  • Uploaded on

Symmetric Multi-modal Intelligent Interactive Development Environment (SMIIDE) Seedling. Update as of September 21, 2009. BAE AIT: Greg Sullivan, Basil Krikeles, Mike Cook MIT CSAIL: Randy Davis, Howie Shrobe, James Oleinik, Kyle Mill.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Symmetric Multi-modal Intelligent Interactive Development Environment (SMIIDE) Seedling' - ashanti

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Symmetric multi modal intelligent interactive development environment smiide seedling

Symmetric Multi-modal Intelligent Interactive Development Environment (SMIIDE) Seedling

Update as of

September 21, 2009

BAE AIT: Greg Sullivan, Basil Krikeles, Mike Cook

MIT CSAIL: Randy Davis, Howie Shrobe, James Oleinik, Kyle Mill

Symmetric multimodal interactive intelligent development environments
Symmetric Multimodal Interactive Intelligent Environment (SMIIDE) SeedlingDevelopment Environments

  • Dramatic reduction in time and cost of constructing software designs and related engineering artifacts

  • Production of higher quality artifacts through interactive design.

  • Metrics to be refined during seedling.

Let’s create a data interface component

Does this process need access to the database?



  • Current Integrated Development Environments (IDEs) are developer-centric, detail-intensive, and error-prone

  • Modality is essentially paper-based and human developer-centric

  • IDEs support limited consistency checking and debugging

  • Multi-modal Mixed-initiative User Interface

    • Supports sketching, gestures, natural language.

    • System can query human for clarification, suggestion, etc.

    • System can take initiative in design and implementation, freeing the human from detailed specification of lower-level details

  • Adaptive Domain Specific Model-Based Software Engineering tools

    • Support evolution of domain “vocabulary”

    • Support diagramming using symbology specific to a domain.


    • Anticipated productivity gains can be demonstrated on well-defined domains.

    • Current natural language technology is sufficient.

  • • Demonstrate program feasibility and develop program structure

    • Prototype combination of symmetric multimodal HCI with model-based code generation technology

    • Determine feasibility of program leading to mixed-initiative high-productivity environments

    • Identify technology gaps and refine DARPA-hard program challenges.

    • Demonstrate on a DoD-relevant domain

    • Survey and position with respect to previous research

Multi-modal User Interfaces can now integrate sketch, gesture,& natural language to capture design, rationale more naturally.

Mixed-Initiative User Interfaces enable symmetric collaboration between human and computer-based designers.

Adaptive, Domain-Specific Model-Based Software Engineering tools enable flexible collaboration at a human level of abstraction.



Symmetric multimodal interaction brings intelligence to model-based development

SMIIDEling Overview


Distribution authorized to U.S. Government Agencies only

Some seedling goals from jim s slides at august meeting
Some seedling goals Environment (SMIIDE) Seedling(from Jim’s slides at August meeting)

  • Arrive at a focused set of technical challenges that form the basis of a new program

  • The purpose of the seedling is to create a new program initiative brief

  • A “Scientific American” statement of the technical goals of the program

  • A clear connection to compelling operational need

  • A convincing case for how this can be achieved

  • The nature of the technical challenges, and which focused challenges will form the basis for a program?

  • How far can we reach in terms of operational capability in the scope of a program?

  • What is the potential operational payoff?

  • What options for demonstrating the concept and measuring success?

SMIIDEling Overview

Outline Environment (SMIIDE) Seedling

  • The Heilmeyer Questions

  • A “Scientific American” statement

  • Compelling operational need

  • Research Topics for proposed program

    • Quick review of topics from August meeting.

    • “Lots of Little Languages”.

    • Rationale Capture via Natural Interaction

    • Related research (“Why now?”)

  • Sample Scenarios

    • Geospatial reasoning in Deep Green

    • Building a wiki for after-action reports

SMIIDEling Overview

1 the heilmeyers
1. The Heilmeyers… Environment (SMIIDE) Seedling

  • What is the problem, why is it hard?

    • Software design, development, and maintenance.

      • Low level, error-prone, complex, hard to verify.

      • Slow to adapt to mission needs.

  • How is it solved today?

    • Code-level “forward engineering”. That is, human-intensive high-to-low level specification with computer “automaton” support for detail tracking (e.g. parsing) and high-to-low level translation (e.g. compilation).

  • What is the new technical idea; why can we succeed now?

    • Natural interaction and DSML interpretation combine to enable mixed initiative interaction, with useful initiative from computer, augmented by captured rationale.

    • Drill down and machine learning enable both computer and human input at all levels of abstraction.

  • What is the impact if successful?

    • Revolutionary increase in productivity for creating software-intensive systems. Enables fast development and deployment of latest technology, rather than traditional slow DoD software acquisition process.

    • Dramatic increase in development / adaptation allows rapid response to mission needs.

  • How will the program be organized?

  • How will intermediate results be generated?

  • How will you measure progress?

  • What will it cost?

SMIIDEling Overview

A scientific american statement of the technical goals of the program
A “Scientific American” statement of the technical goals of the program

  • Every day, countless conversations take place between software engineers in front of white boards. The engineers take turns grabbing markers, and talk while drawing. In response to questions from each other, the engineers elaborate or erase parts of their diagrams. The interactions are highly verbal, and the drawings include text, traditional software diagrams, and ad-hoc notations made up on the spot.

  • Now, what if one of those engineers was a computer?

  • The goal of the SMIIDE program is investigate whether the combination of several cutting edge research areas can enable computer-based agents to become active participants in the software design, development, and maintenance process.

  • The computer should be able to ask relevant questions, suggest useful approaches and designs, and thereby substantially contribute to the implementation of the eventual system.

Research Areas: Sketch recognition, Natural Language, Rationale Capture, Domain Specific Modeling, Dialogue management.

SMIIDEling Overview

Compelling operational need
Compelling Operational Need of the program

  • The problem: in a fast-changing world, it’s easy to miss the mission window

  • Response: Rapidly tailor systems to mission needs

    • Can leverage rationale captured in natural modalities.

      • Why were certain decisions made?

      • What were assumptions?

  • Increased speed of development while maintaining correctness

  • Decreased costs

What current operational needs are addressed by a dramatic increase in speed and flexibility?

SMIIDEling Overview

Research topics identified during august meeting
Research Topics identified during August meeting of the program

  • Grounding complex abstract diagrams in sufficient meaning to support the design task;

  • Determining how complex specifications can be segmented in ways that are functionally meaningful for reasoning and dialog;

  • Determining what kind of human-computer interactions are needed at any given moment and mediating that interaction effectively;

  • Determining the HCI principles that lead to productivity enhancements not attainable with model-based software engineering tools alone (including the advanced versions produced in the current DMT-SWP program).

  • Is there a body of kn about MBSE applicable across domains?

    • Dev of DSMLs suggest any common body of language, lexicon,…

    • How wide a variety of specification forms to consider? tabular, text, graphical

  • Graphical models vs. behavioral specifications? How do we get the benefits of graphical structural models for behavioral specs?

SMIIDEling Overview

Core research topic related families of overlapping dsmls
Core Research Topic: Related Families of Overlapping DSMLs of the program




  • Benefits of DSLs:

    • Small language decreases errors.

    • More abstraction increases productivity, improves analysis

    • DSMLs capture expertise of domain experts

  • How to get coverage / generality?

    • Lots of DSLs

    • Overlaps in domains gain coverage by transitivity.

    • Evolution of DSLs

  • Generators / Interpreters / Transforms

    • Provide overlap by translation

    • Often high-to-low level (where “level” is loosely defined).

    • Generator logic can be hand-generated or partially learned.


Course of Action


Correspondence, Transformations


UML classdiagrams

Correspondence, Transformations




SMIIDEling Overview

Core research topic natural rationale capture
Core Research Topic: Natural Rationale Capture of the program

  • Can natural interaction make rationale capture (nearly) painless?

    • Why were certain decisions made?

    • What were assumptions?

  • Can captured rationale provide the basis for “parametric design” of software?

SMIIDEling Overview

Rationale capture rationale
Rationale Capture - Rationale of the program

  • Every software project involves thousands of design decisions

    • Most are routine (i.e., obvious to an experienced programmer); many crucial ones are not

  • Writing/modifying/debugging/ software requires constantly re-absorbing those decisions.

  • This would be orders of magnitude easier if all the program authors were always there to explain how they made those decisions and why.

  • Rationale capture can achieve this effect.

SMIIDEling Overview

Rationale capture technical challenges
Rationale Capture – Technical Challenges of the program

  • Why isn’t rationale captured now?

    • Cost is too high (writing documentation is a lot of work)

    • Task is aversive (writing documentation is unpleasant work)

    • Task is disruptive (writing documentation interrupts design)

    • The value equation is out of whack:

      • The person incurring the cost is (almost) never the one who gains the benefit.

      • Cost is now, benefit is in the future.

  • Consequences: documentation of small but crucial decisions is deferred.

Can we change the equation by drastically lowering the cost and making it more like talking to another designer?

SMIIDEling Overview

Core research topics sketch understanding symmetric multimodal interaction
Core Research Topics: Sketch Understanding, Symmetric Multimodal Interaction

  • Sketch understanding

    • grounding complex abstract diagrams in sufficient meaning to support the design task;

    • structure (e.g., data structure) is easy to draw, but a lot of software is about process, behavior

      • can we find ways to draw behavior that feel as natural?

  • Symmetric multimodal interaction requires an intelligent observer, e.g., knowing what questions to ask and when

    • How task- and domain-specific is that knowledge?

    • Is there a body of knowledge about MBSE applicable across domains? Or is the problem AI-complete?

SMIIDEling Overview

Central hypotheses
Central Hypotheses Multimodal Interaction

  • Natural interaction + modestly intelligent observer = drastic increase in agility and lower cost

  • Domain specificity enables intelligence of computer-based observer.

  • Evolvable and overlapping languages enables generality

SMIIDEling Overview

How far can we reach in terms of operational capability in the scope of a program
How far can we reach in terms of operational capability in the scope of a program?

  • What to rule out?

    • “Programmer’s common sense”

      • Domain expertise encoded in DSML “smarts”.

      • Mixed initiative: human in the loop.

  • System must be “smart about something” but cannot be smart about everything.

SMIIDEling Overview

What options for demonstrating the concept and measuring success
What options for demonstrating the concept and measuring success?

  • Prototype and storyboards for demonstrating concept.

SMIIDEling Overview

Metrics success?

  • Ultimate metric: order of magnitude faster with system – allows missions / capabilities that simply weren’t available before.

  • # of lines of code automated versus hand written.

  • Some measure of complexity of code generated. In other words, abstraction + automation results in less quantity and lower complexity over course of task.

SMIIDEling Overview

Program structure
Program Structure success?

  • Individual pieces, with an integrator

  • Or entire systems from each team?

SMIIDEling Overview

Related research
Related Research success?

  • Sketch understanding

    • MIT (Davis), UWash (Landay), UC Riverside (Stahovich), Illinois (Forbus)

  • Dialogue management

    • Grosz (Harvard), Sidner (BAE)

  • DSMLs

    • Karsai et al. (Vanderbilt), Metacase (Kelly, Tolvanen), Gray (Alabama), Sprinkle (Arizona).

    • See

  • Natural Language

  • Mixed initiative

SMIIDEling Overview

Random thoughts
Random Thoughts success?

  • Community aspect?

    • related to Framework Adoption Seedling

    • learning over many interactions with same and different users.

  • Related to Bootstrapped Learning?

    • Teaching computer to program.

    • Might be relevant to helping SMIIDE system learn.

SMIIDEling Overview