1 / 65

Automated Generation of Visual Discourse

Automated Generation of Visual Discourse. Michelle X. Zhou Department of Computer Science Columbia University New York, NY 10027 Supported in part by DARPA Contract DAAL01-94-K-0119, New York State Science and Technology Foundation, NSF Grant ECD-88-11111, and ONR Contract N00014-97-1-0838.

malvina
Download Presentation

Automated Generation of Visual Discourse

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automated Generation of Visual Discourse Michelle X. Zhou Department of Computer Science Columbia University New York, NY 10027 Supported in part by DARPA Contract DAAL01-94-K-0119, New York State Science and Technology Foundation, NSF Grant ECD-88-11111, and ONR Contract N00014-97-1-0838

  2. Automated Visual Design Problem • Designing effective visual presentations is difficult and costly • Designing customized visual presentations in a timely manner is more difficult Approach Develop computer techniques to automate visual design process

  3. Input ??? Automated Visual Design

  4. discrete Previous Work • Single displays APT Mackinlay 86 SAGE Roth & Mattis 91 ANDD Marks 91 • A series of displays APEX Feiner 85 IBIS Seligmann 91 WIP Andre et al. 93 . . . . . .

  5. Scale+Move+... Open Rotate Our Goal: Visual Discourse Design A series of connected displays Open Rotate Scale Move . . .

  6. IMPROVISE • Computer network • management • Patient medical • record summary Design Foundation • Data characterization • Visual task hierarchy • Presentation design • language • Inference paradigm System Design • Knowledge base • Inference engine • Visual realizer • Interaction handler Thesis . . . . . . Visual Discourse Modeling • Coherence • Versatility • Interactivity Thesis Work System Development General Approach

  7. Thesis . . . . . . IMPROVISE • Computer network • management • Patient medical • record summary Design Foundation • Data characterization • Visual task hierarchy • Presentation design • language • Inference paradigm System Design • Knowledge base • Inference engine • Visual realizer • Interaction handler Thesis Work Visual Discourse Modeling • Coherence • Versatility • Interactivity System Development General Approach

  8. Visual Discourse Modeling • Coherence • Versatility • Interactivity Continuity Consistency Unity Wide range of information Wide variety of visual media/techniques Interruptible Responsive

  9. IMPROVISE • Computer network • management • Patient medical • record summary System Design • Knowledge base • Inference engine • Visual realizer • Interaction handler Thesis . . . . . . Thesis Work Design Foundation • Data characterization • Visual task hierarchy • Presentation design • language • Inference paradigm Visual Discourse Modeling • Coherence • Versatility • Interactivity System Development General Approach

  10. Output + Design Foundation: Design Process Input Presentation Data Presentation Context Design Engine Presentation Intents Design Knowledge

  11. Output + Design Foundation Input Data Characterization Presentation Data Presentation Context Design Engine Presentation Intents Design Knowledge

  12. …. …. ….. …. Data Characterization Data Visual Elements Goal Previous Work Characterizing Quantitative data Mackinlay 86 Roth & Mattis 90 Presentation-related data properties Characterizing Qualitative data Arens et al. 93

  13. …. …. ….. …. Our Approach: Characterizing Heterogeneous Data Data Dimensions Domain Type Attribute Relation Role Sense (Jones (is-a PATIENT) (type ATOMIC) (property (Form …)...) (relation (rel …)...) (role LOCATE) (sense SYMBOL)) Presentation-related data properties

  14. Atomic Composite Type Entity Concept Measurement Event Domain Form Material Location Transience Importance Composite Attributes Attributes Ordering Scalability Continuity FD Constituency Attribute Enumeration Relation Associate Background Categorize Cluster Correlate . . . Role Label List Plot Symbol Portrait Sense Data Characterization Taxonomy

  15. Output + Design Foundation Input Data Characterization Presentation Data Context Modeling Presentation Context Design Engine Presentation Intents Design Knowledge

  16. Context Modeling Goal Context Information Visual Techniques Previous Work Situation space Friedell 83 Display categories Mackinlay 86 Our Approach Audience Occasion Environment

  17. Output + Design Foundation Input Data Characterization Presentation Data Context Modeling Presentation Context Design Engine Presentation Intents Design Knowledge Intent Modeling

  18. Presentation Intents Wehrend & Lewis 90 Casner 91 Marks 91 Roth & Mattis 91 Ignatius & Senay 94 Visual Techniques Hunter 87 Levin 87 Maybury 93 Seligmann 93 Sutcliffe et al. 94 Intent Modeling Goal Presentation Intents Visual Techniques Previous Work

  19. Achieve Imply Achieve Imply Our Approach: Visual Task Characterization Presentation Intents Visual Techniques Visual Tasks (Visual Effects) Enlarge Highlight Zoom Search Elaborate Focus Abstract Visual Technique

  20. Generalize Merge Identify Name Portray Individualize Profile Locate Position Situate Pinpoint Outline Rank Time Reveal Expose Itemize Specify Separate Switch Background Associate Colocate Connect Unite Attach Categorize MarkDistribute Cluster Outline MarkDistribute Correlate Plot MarkCompose Compare Differentiate Intersect Distinguish MarkDistribute Isolate Emphasize Focus Isolate Reinforce Encode Label Symbolize Quantify Iconify Portray Tabulate Plot Trace Structure Map Visual Task Taxonomy

  21. Output + Design Foundation Input Data Characterization Presentation Data Context Modeling Presentation Context Design Engine Design Knowledge Modeling Presentation Intents Design Knowledge Intent Modeling

  22. Design Knowledge Modeling Goal Computational representation of design knowledge Previous Work Visual formalisms Marks 91; Lohse et al. 94 Visual techniques Friedell 84; Seligmann 93; Keller&Keller 94 Visual design principles Winn&Holliday 82; Bertin 83; Mullet&Sano 95 Mackinlay 86; Ignatius&Senay 94; Tufte83, 90, 97

  23. Our Approach: Presentation Design Language • Visual objects Represent various visual formalisms • Visual techniques Compose/manipulate visual objects • Visual design principles Guide the visual object composition and manipulation

  24. + red I New York love Visual Object Representation • Syntax • Semantics • Pragmatics Patterns/compositions Meanings human heart Specific meanings

  25. Discourse Visual Lexicon (atomic objects only) Visual Frames Visual Structures Boston Visual Unities Visual Primitives Visual Object Hierarchy Static/Dynamic Tables Charts Diagrams Shape/Color/Size/Orientation/...

  26. Examples Visual Techniques (DesignTableChart (is-a FormationTech) (operands ?data-obj ?table) . . .) • Categorized by function Formation Transformation Camera (Move (is-a TransformationTech) (operands ?obj) (source) (destination) (startTime) (endTime)) • Categorized by usage Primitive Composite (SetCamera (is-a CameraTech) (operand ?camera) (position) (orientation) . . .)

  27. Visual Design Principles • Expressiveness rules • Effectiveness rules • Comprehensiveness & distinctiveness • Generality & discreteness • Integrity x

  28. Visual Design Principles • Expressiveness rules • Effectiveness rules • Comprehensiveness & distinctiveness • Generality & discreteness • Integrity • Accuracy & clarity • Appropriateness • Immediacy • Continuity • Consistency • Unity

  29. Output + Design Foundation Input Data Characterization Presentation Data Inference Modeling Context Modeling Presentation Context Design Engine Design Knowledge Modeling Presentation Intents Design Knowledge Intent Modeling

  30. Inference Modeling Goal Flexible and efficient design method Previous Work Constructive design Mackinlay 86; Roth & Mattis 90; Marks 91 Parametric design Zdybel et al. 81; Robertson 91; Ignatius&Senay 94

  31. Our Approach: Hybrid Inference Paradigm • Constructive synthesis • Parametric synthesis Planning presentations from scratch Efficiently create visual models for atomic data objects A least-commitment constraint-based approach

  32. Visual Discourse Collection of frames Plan Collection of actions Visual Techniques Visual/Domain Objects Design Principles Operators Objects Constraints + PREVISE (Zhou 97) DPOCL (Young et al. 95) SIPE (Wilkins 88) Planning Elements & Features Practical hierarchical-decomposition partial-order planning

  33. PREVISE (Zhou97) Top-down strategy Action decomposition +Object decomposition Variables +Enriched Variables Temporal reasoning +Spatial reasoning + DPOCL (Young et al. 95) SIPE (Wilkins88) Extended Features +Dynamic numerical constraints +Domain heuristics

  34. Thesis . . . . . . IMPROVISE • Computer network • management • Patient medical • record summary Design Foundation • Data characterization • Visual task hierarchy • Presentation design • language • Inference paradigm System Development General Approach Thesis Work System Design • Knowledge base • Inference engine • Visual realizer • Interaction handler Visual Discourse Modeling • Coherence • Versatility • Interactivity

  35. Knowledge Base • Domain data • Situation data • Visual data • Meta data • Interaction Handler • Present interactivity • Control interactivity • Inference Engine • PREVISE • Visual lexical chooser • Visual Realizer • Interface language design System Framework

  36. Thesis . . . . . . Design Foundation • Data characterization • Visual task hierarchy • Presentation design • language • Inference paradigm System Development General Approach Thesis Work IMPROVISE • Computer network • management • Patient medical • record summary System Design • Knowledge base • Inference engine • Visual realizer • Interaction handler Visual Discourse Modeling • Coherence • Versatility • Interactivity

  37. IMPROVISE (Illustrative Metaphor PROduction in VISual Environments) • Stand-alone graphics system Computer network management • Graphics generator in a multimedia presentation system Healthcare

  38. Messenger Architecture Task Analyzer Interaction Handler Presentation Planner Content Planner Designer Chooser Organizer Stylist Coordinator Knowledge Base Converter Renderer

  39. Example: Present Patient Information to Nurse

  40. Sample Visual Discourse Representation (Discourse1 (is-a VISUAL-DISCOURSE) (frames [Frame1] [Highlight2] . . .)) (Frame1 (is-a VISUAL-FRAME) (frameElements [StructDiag1]) (startTime 0.0) (endTime -1.0)) (Highlight2 (is-a Highlight) (operands [Table1-demo]) (style MARKER) (color . . .) (startTime 5.0) (endTime 8.0) . . .) (StructDiag1 (is-a STRUCTURE-DIAGRAM) (heading [Table1-Demo) (core [Unity1-body]) (elements [Unity2-device1] . . .) . . .)

  41. (Jones-info (is-a CONCEPT) (type COMPOSITE) (convey [Jones] [Jones-demo] [Swan-Ganz] . . .) . . .) (Jones-demo (is-a CONCEPT) (type COMPOSITE) (convey [Jones-name] . . .) (role Identify) (sense LIST) . . .) (Jones (is-a PATIENT) (type ATOMIC) (role LOCATE) (sense SYMBOL) . . .) (Swan-Ganz (is-a DEVICE) (type ATOMIC) (convey [sw-name] [sw-loc] [sw-content]) (sense SYMBOL). . .) Input: Presentation Data (Jones-name (is-a NAME-ATTR) (value “S. Jones”) (role IDENTIFY) (sense LABEL). . .)

  42. Input: Presentation Context (Context-info (audience (identity NURSE) . . .) (occasion (presentation ON-LINE) (medium VISUAL-SPEECH) . . .) (environment (display (color COLOR) (size . . .) . . .) (platform (cpu SGI R4400) . . .)))

  43. Input: Presentation Intent & Visual Task Formation Intent: Summarize<Jones-info> Structure<Jones-info> Identify<Jones-info, Jones-demo> Locate<Jones-line, Jones> Symbolize<Jones-line> Name<Jones-demo, Jones-name> Itemize<Jones-demo> Name<Jones-line, line-name> Associate<Jones-line, line-content>

  44. Fulfill Visual Tasks Intent: Summarize<Jones-info> DesignStructureDiagram <Jones-info, ?diag> Structure<Jones-info> Identify<Jones-info, Jones-demo> Locate<Jones-line, Jones> Symbolize<Jones-line> Itemize<Jones-demo> Name<Jones-demo, Jones-name> Name<Jones-line, line-name> Associate<Jones-line, line-content>

  45. Structure Diagram Patient Info

  46. Fulfill Visual Tasks Intent: Summarize<Jones-info> DesignStructureDiagram <Jones-info, ?diag> Structure<Jones-info> DesignVisRep<Jones-demo, ?h> DesignVisRep<Jones, ?core> DesignVisRep<Johns-line, ?el> Identify<Jones-info, Jones-demo> Locate<Jones-line, Jones> Symbolize<Jones-line> Itemize<Jones-demo> Name<Jones-demo, Jones-name> Name<Jones-line, line-name> Associate<Jones-line, line-content>

  47. Heading Identify Structure Diagram Element Symbolize Device Device Core Locate Element Symbolize Element Symbolize Patient Device Device Element Symbolize Device Element Symbolize Demographics

  48. Fulfill Visual Tasks Intent: Summarize<Jones-info> DesignStructureDiagram <Jones-info, ?diag> Structure<Jones-info> DesignVisRep<Jones-demo, ?h> DesignVisRep<Jones, ?core> DesignVisRep<Johns-line, ?el> Identify<Jones-info, Jones-demo> Locate<Jones-line, Jones> Symbolize<Jones-line> DesignTable<Jones-demo, ?t> DesignVisRep<Jones-name, ?th> Itemize<Jones-demo> Name<Jones-demo, Jones-name> Name<Jones-line, line-name> Associate<Jones-line, line-content>

  49. Heading (Table) Structure Diagram Element Symbolize Device Device Element Symbolize Element Symbolize Device Device Element Symbolize Device Element Symbolize Core Locate Patient

  50. DesignVisRep<jones-name, ?table-heading> DesignVisRep<jones-identification, ?table-cell1> DesignVisRep<jones-medHistory, ?table-cell2> DesignVisRep<jones-operation, ?table-cell3> DesignTable <jones-identification, ?table11> DesignTable <jones-operation, ?table13> DesignTable <jones-medHistory, ?table12> DesignVisRep<jones-age,?table11-cell1> DesignVisRep<jones-gender,?table11-cell2> DesignVisRep<jones-mrn, ?table11-cell3> DesignVisRep<op-name,?table13-cell1> DesignVisRep<op-surgeon, ?table13-cell2> DesignWord<op-name, ?op-name-rep> DesignWord<op-surgeon, ?surgeon-rep> . . . DesignWord<jones-age, ?age-rep> DesignWord<jones-gender, ?gender-rep> DesignWord<jones-mrn, ?mrn-rep> DesignTable<jones-demo, ?table>

More Related