The General
Download
1 / 35

The General Temporal Workbench A Universal System for Exploring Time-based Phenomena Donald Byrd - PowerPoint PPT Presentation


  • 79 Views
  • Uploaded on

The General Temporal Workbench A Universal System for Exploring Time-based Phenomena Donald Byrd Woodrow Wilson Indiana Teaching Fellow and Research Technologies & School of Informatics Indiana University Bloomington 2 May 2013. Introduction (1): Time Is of the Essence.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The General Temporal Workbench A Universal System for Exploring Time-based Phenomena Donald Byrd' - roger


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The general temporal workbench a universal system for exploring time based phenomena donald byrd

The General Temporal Workbench

A Universal System for Exploring Time-based Phenomena

Donald Byrd

Woodrow Wilson Indiana Teaching Fellow

and

Research Technologies & School of Informatics

Indiana University Bloomington

2 May 2013


Introduction 1 time is of the essence
Introduction (1): Time Is of the Essence

  • Gandini Juggling’s Mozart (Symphony 25, 1st mvmt) performance

  • How created? How did jugglers learn it?

  • I don’t know, but I do know was a huge pain!

  • Likewise for doing almost anything w/ complex temporal phenomena

  • …and timescale can be yoctoseconds or billions of years

  • Natural phenomena

    • Inflation of early universe, chemical reactions, animal movement, weather, illnesses, climate; development of individuals, of species, of geographic features, & of galaxies

  • Artificial & cultural phenomena

    • Movies, behavior of computer programs, football games, multimedia shows, operas, political crises, wars, development of civilizations

1 May 2013


Introduction 2
Introduction (2)

  • Complex temporal phenomena difficult to study or (for performances) to create, because don’t “sit still”

  • Can “play” repeatedly, but at expense of context

  • Usual answer: visualization

  • Good start, but we can (and need to) do much better than existing systems

  • For music, General MusicVisualizer (GMV)

  • Music ex: Hendrix/Star Spangled Banner: how to notate?

  • Musical audio: Shakuhachi solo: how to segment? (Nina Fales)

    • Might be a good solution to notating Hendrix!

    • Studying speech or animal sounds is a similar problem

3 July 2009


Gmv demo segment shakuhachi solo
GMV “demo”: Segment shakuhachi solo

  • Scenario: music-informatics researcher or ethnomusicologist comparing two audio-segmentation algorithms

  • …or composer comparing input & output of synthesis programs

20 Apr. 2009


The problem for multimedia
The Problem for “Multimedia”

  • “Multimedia” = performance w/ music & sync’ed movement (dance, etc.) or visuals (fireworks, etc.)

  • Requirements

    • 1. People need several ways to “look” at an instance simultaneously

    • 2. People need to be able to compare instances

    • 3. Has specialized notation (at least for music)

    • 4. Media: video, audio, still images, text

      • Audio independent of video is often important

1 May 2013


The solution for multimedia
The Solution for Multimedia

  • General MusicVisualizer => General Multimedia Workbench (GMW)

  • Early prototype of GMW: AWB Plus

    • Based on EVIA Annotator’s Workbench

    • Java program, developed for ethnography

  • Queen of the Night aria, in “GMW”

  • Fireworks show with music, in AWB Plus

28 Mar. 2010


Gmw screenshot queen of the night aria
GMW “Screenshot”: Queen of the Night aria

  • Scenario: singer (or conductor) comparing videos of performances to learn role in opera, musical, etc.

  • …or stage director, choreographer, or lighting designer comparing previous versions to own ideas

  • …or scholar studying performances (perhaps juggling w/ music!)

4 Apr. 2009


Fireworks show with music in awb plus
Fireworks show with music, in AWB Plus

  • From video & still photos of show by Mike Durbin

    • All timestamped => (semi)automatic alignment

  • Music added afterwards!

28 Mar. 2010


The general problem
The General Problem

  • Long-standing, difficult problems in all fields

    • Affect all uses: research, creation, “aesthetics”, organization

  • …plus deluge of data in many fields

    • Even arts & humanities are getting lots of hard data

    • Data promises to help, but not much help so far!

  • Need insight, not data; how to get there?

    • Widely recognized as important goal

1 May 2013


Towards a solution 1
Towards a Solution (1)

  • Review: multimedia features

    • 1. People need several ways to “look” at instance simultaneously

    • 2. People need to be able to compare instances

    • 3. Has specialized notation (at least for music)

    • 4. Media: video, audio, still images, text

  • Observations

    • Multimedia features #1 & 2 are true of every (and 3 & 4 of many) complex temporal phenomenon

    • Every temporal phenomenon is complex

    • People often want to relate phenomenon of one type to another

1 May 2013


Towards a solution 2
Towards a Solution (2)

  • The cross-discipline argument

    • Problems in all fields have much in common => potential for synergy

  • But people rarely share ideas or software across disciplines

    • Issue of “disjoint technical vocabulary/literature” (cf. Swanson 1988)

    • Ex: Are there periodic patterns in public opinion? => political scientists should try Fourier analysis, but…

  • The research vs. creation vs… argument

    • Problems of researchers, creators, etc. in a field have much in common

  • => A general temporal system could greatly reduce “reinventing the wheel” or doing without one

1 May 2013


The general solution
The General Solution

  • General temporal system applicable to very many disciplines is possible

  • General temporal system applicable to all types of use is possible

  • GMusicW needs so much flexibility, a GAnythingW wouldn’t need much more

  • Oh, yeah? With what magic?

  • The magic of Software Architecture!

1 May 2013


Architecture for a general system 1
Architecture for a General System (1)

  • Main tasks for framework: (temporal) synchronization, UI support

  • Software for common audio & video uses:

15 Mar. 2009


Architecture for a general system 2
Architecture for a General System (2)

For sequential art & movies based on sequential art (John Walsh):

15 Mar. 2009


General temporal toolkit workbench gtt w
General Temporal Toolkit/Workbench (GTT/W)

  • “Temporal”, not just “Multimedia”

  • Supports multiple: coordinated, editable, interactive visualizations & sonifications (eventually “tactilizations”, “olfactizations”?)

  • …of multiple instances

  • …of any combination of temporal phenomena

  • …plus data mining & analysis (Solution 2)

  • For all applications (research, creative, other)

  • Myosin V, in AWB Plus

5 Mar. 2010


Important feature 1 better human machine partnerships
Important Feature 1: Better Human/Machine Partnerships

Figure slightly modified from Yu et al (2008)

  • Integrate info visualization & analysis/data mining (Shneiderman 2002; Yu et al 2008) => closed loop: use visual perception to generate hypotheses for analysis; present results of analysis visually

    • “HCIR”, visual analytics

  • …or substitute synthesis for analysis, e.g., for composers

7 Apr. 2009


Important feature 2 3 allow all sensory modes automatic synchronization
Important Feature 2 & 3: Allow All Sensory Modes; Automatic Synchronization

  • Sensory Modes

    • Visual: visualization is most generally useful, but not the only answer

    • Auditory: sound is central for many applications

      • sonification is surely valuable for some non-audio phenomena

    • Tactile: important for the blind

    • Other (olfactory): important for ??

    • Don’t rule out support for all sensory modalities

  • Automatic Synchronization

    • Of audio (at least) has made great progress in last 10 years

2 May 2013


Myosin v in awb plus
Myosin V, in AWB Plus Synchronization

  • From nanobioscience work by Craig & Linke

  • Myosin V crawling along a strand of acetin

28 Mar. 2010


Big issue time representation
Big Issue: Time Representation Synchronization

  • Resolution/precision

    • Cosmology: Planck time to quadrillions(?) of years

  • Precision & handling of uncertainty

    • Historical forms: “1685?”, “fl. mid-15th century”, etc.

    • Damaged sources: “25th of [unreadable] 1743”, etc.

    • Handle with Probability Density Functions?

  • Absolute & relative time

    • Calendars are incredibly messy

  • Design of universal time representation is well along

1 May 2013


Huge issue getting off the ground 1
Huge Issue: Getting Off the Ground (1) Synchronization

  • AWB Plus prototype, based mostly on EVIA Annotators Workbench; some of Variations

    • may also use CIShell, Chen Yu’s system for “visual mining of multimedia data”

    • all from IUB

  • Other possible open-source starting points

    • Sonic Visualiser, SyncPlayer, SIMILE Timeline, etc.

  • Connections to general tools for nontemporal visualization

    • Network Workbench (Katy Börner/IU SLIS): networks

    • Google Map API: “space” (surface of the earth)

  • Connections to semi-general tools

    • Ex: Max/MSP, Pd: musical audio, video

    • Ex: imagej for image display/processing

1 May 2013


Getting off the ground 2
Getting Off the Ground (2) Synchronization

  • Support from communities of users is almost essential

  • Have seen strong interest for:

    • Ethnomusicology field work

    • Opera scholarship

      • “Could revolutionize study of opera”

    • Conducting

    • Teaching electronic music

    • Study of computer-aided musical improvisation

    • Study of comic books/movies

    • Research on artificial life

    • Software engineering, research software

    • Nanoscience/nanotechnology

    • Study of French

1 May 2013


Conclusions
Conclusions Synchronization

  • How do I know applications are realistic?

    • Many probably aren’t, but many, many possibilities exist!

    • Have ca. 30 usage scenarios, ca. half written/endorsed by experts

    • Some examples (all experts in respective fields)

      • Ruth Stone: ethnomusicology field work

      • Philip Gossett, Nick Cook: study of music/opera performance

      • Paul Nadler: conducting

      • John Gibson: teaching electronic music

      • Elaine Chew: annotating video of computer-aided musical improvisation

      • John Walsh: comic books/movies

      • Larry Yaeger: artificial life

      • Fabian Fagerholm, Ben Kovitz: software engineering, research S/W

      • Amar Flood: nanoscience/nanotechnology

    • Personal knowledge/experience for a few

1 May 2013


The general temporal workbench a universal system for exploring time based phenomena donald byrd
End Synchronization

  • Thanks to Eric Isaacson, Geoff Chirgwin, Will Cowan, Allen Winold, Paul Sturm, Ben Kovitz, Craig Stewart, Eric Wernert

  • THE END

1 May 2013


Extra slides
Extra Slides Synchronization

  • Following slides (with background like this one) are just in case…

rev. 18 Feb. 2009


Good design for music can be good for many things
Good Design for Music Can Be Good for SynchronizationMany Things

  • Cf. “Why Studying Music is Both Difficult and Important” (Byrd 2009)

  • Music is an art => people use elements in unusual ways

  • Music is a performing art =>performances & symbolic representations

  • Much music hascomplex synchronization requirements

  • Music involves many different instruments, often in groups. Leads to:

    • Arrangements/transcriptions for other instruments

    • Versions for players with different levels of skill

    • Notation may represent sounds or actions

  • Music is often combined with text via singing, narration, etc.

  • Music is extremely popular, so:

    • Some works exist in many versions, arrangements for different ensembles, etc.

    • Handling challenges is important, even on purely economic grounds

rev. 18 Feb. 2009


Hci searching browsing visualization
HCI: Searching, Browsing, & Visualization Synchronization

  • Visualization is essential for browsing, merely helpful for searching

  • In browsing, user finds everything; the computer just helps

  • Browsing is obviously good because it gives user more control, but few systems emphasize it. Why?

    • “Users are not likely to be pleasantly surprised to find that the library has something but that it has to be obtained in a slow or inconvenient way. Nearly all items will come from a search, and we do not know well how to browse in a remote library.” (Lesk, p. 163)

  • For “and”, read “as long as”

  • Searching is more natural on computer, browsing in real world

    • Effective browsing takes very fast computers—widely available now

    • Effective browsing has subtle UI demands

  • Cf. HCIR, visual analytics, visual searching, etc.

7 Mar. 2009


Why juggling who cares
Why juggling? Who cares? Synchronization

  • A surprising domain, but realistic

    • Features 1 & 2 apply

    • Feature 3 applies in part: has established (though not graphic) notation, “siteswap”

  • Many juggling programs available

  • GTW framework has support for:

    • Control of tempo, including pausing or going backwards

    • UI for (temporal, not spatial) zooming in on details

    • Synchronization of multiple videos and/or animations

    • Framework for auto. synchronization

    • Framework for combining independent visualizations

  • Animal motion in general is much more complex => more need for GTW!

    • Ex: dancing (Labanotation, etc.)


Structure in basic representations of music audio
Structure in Basic Representations of Music & Audio Synchronization

Western Music Notation: very complex, irregular structure; some parts well-defined, some not—and what’s well-defined isn’t well-defined

Audio: no explicit structure

MIDI: simple, regular, well-defined structure

10 Feb. 09


Basic representations of music audio
Basic Representations of Music & Audio Synchronization

1. Audio (e.g., CD, MP3): like speech

2. Time-stamped Events (e.g., MIDI file): like unformatted text

3. Music Notation: like text with complex formatting

  • Time scales of graphs: #1, milliseconds; #2 & 3, seconds

  • Essential difference among forms: “knowledge representation” = explicit structure

10 Feb. 09


Isn t it a mistake to use music notation this way
SynchronizationIsn’t it a mistake to use music notation this way?”

  • Chris Raphael’s question about Hendrix transcription

  • It’s obviously useful: easy to find phrases, “Taps”, etc.

  • …but seriously misleading in places

  • But CMWN is “always” misleading!

  • Is it useful enough to justify danger of misleading?

  • Knowledge representation has inevitable bias (Davis et al 1993); notation has more bias (Wiggins et al 1993)

  • Fundamental issue of transcription in ethnomusicology

  • Conclusion: use it, but be careful

    • Cf. my “Logician General’s Warning” on classification

    • …in fact, transcribing requires classifying constantly

12 Feb. 09


Sequential art movie the hard goodbye 1
Sequential-Art/Movie: The Hard Goodbye (1) Synchronization

  • From Frank Miller’s “Sin City” series

  • John Walsh (SLIS): want to compare comics, movies of them, etc.

18 Feb. 2009


Sequential art movie the hard goodbye 2
Sequential-Art/Movie: The Hard Goodbye (2) Synchronization

  • From Frank Miller’s “Sin City” series

  • John Walsh (SLIS): want to compare comics, movies of them, etc.

18 Feb. 2009


Types of visualizations of music and more
Types of Visualizations of Music (and more) Synchronization

  • Is visualization static or dynamic?

    • Dynamic = time represented by time

    • Static = time represented by space

  • What features are visualized?

  • What basic representation? Audio, symbolic, both?

    • Easy to generalize to plays (score = script) & other text phenomena, dance, etc.

18 Feb. 2009


Types of visualizations of music and more1
Types of Visualizations of Music (and more) Synchronization

  • Hendrix example uses coordinated visualizations

    • Generalization of parallel, aligned, synchronized, etc.

  • How are multiple visualizations coordinated?

    • Parallel panes of a single window

    • Superimposed in a single window

    • Separate coordinated windows

  • Forms 1 & 2 apply directly to audio (incl. sonification)

  • Easy to interpolate between forms 1 & 2

    • Categories in the real world are rarely discrete

26 Feb. 2009


The ultimate music system
The Ultimate Music System Synchronization

  • Original goal: visualizer that can do anything with music

    • Handle any no. & combination of visualizations

    • Static visualizations: audio, any kind of notation, structural diagrams, etc.

    • Dynamic visualizations: video, etc.

    • Automatic (or near-automatic) synchronization

    • Support OS-level technologies (QuickTime, etc.)

    • Easy-to-learn UI allowing high degree of control

      • User may want frequent extreme zoom changes => help with

  • If it could do all that, should be useful for lots (domains with >=2 Features) besides music!

20 May 08