evaluating visualizations using a taxonomic guide l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
EVALUATING VISUALIZATIONS : USING A TAXONOMIC GUIDE PowerPoint Presentation
Download Presentation
EVALUATING VISUALIZATIONS : USING A TAXONOMIC GUIDE

Loading in 2 Seconds...

play fullscreen
1 / 47

EVALUATING VISUALIZATIONS : USING A TAXONOMIC GUIDE - PowerPoint PPT Presentation


  • 239 Views
  • Uploaded on

EVALUATING VISUALIZATIONS : USING A TAXONOMIC GUIDE By E. MORSET, M. LEWIS & K. A. OLSEN PRESENTORS CHANNA P. WITANA C ALVIN O R CONTENT Introduction Visual and domain tasks Methodology Tasks Results Discussion Conclusion INTRODUCTION Previous Papers published;

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'EVALUATING VISUALIZATIONS : USING A TAXONOMIC GUIDE' - sandra_john


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evaluating visualizations using a taxonomic guide
EVALUATING VISUALIZATIONS : USING A TAXONOMIC GUIDE

By

  • E. MORSET, M. LEWIS & K. A. OLSEN
  • PRESENTORS
    • CHANNA P. WITANA
    • CALVIN OR
content
CONTENT
  • Introduction
  • Visual and domain tasks
  • Methodology
  • Tasks
  • Results
  • Discussion
  • Conclusion
introduction
INTRODUCTION
  • Previous Papers published;
    • Morse, E. & Lewis, M. (1997).

Why information retrieval visualizations sometimes fail, in Proceeding of the 1997 IEEE International Conference on Systems, Man, and Cybernetics, Oct. 12-15, Orlando, FL

    • Morse, E., Lewis, M., Korfhage, R., and Olsen, K. (1998). Evaluation of text, numeric and graphical presentations for information retrieval interfaces: User preference and task performance measures.Proceedings of the 1998 IEEE International Conference on Systems, Man, and Cybernetics, Oct 12-14, San Diego, CA, 1026-1031
information retrieval visualization systems
Information Retrieval Visualization Systems
  • Bead (Chalmers, 1996)
  • InfoCrystal (Spoerri, 1993)
  • BIRD
  • GUIDO
  • VIBE

These have been developed as visual information exploration tools to aid in retrieval tasks.

in tile bars hearts 1995
In TILE BARS (Hearts, 1995)
  • Paragraphs on X-axis
  • Query items on Y-axis
  • Each query term tile is shaded according to how well the paragraph matches the query term.
  • By glancing the Tile Bar a user can see which query terms match, most relevant sections, distribution and coincidence of topics throughout the document.
in vibe
In VIBE
  • VIBE represents query terms as moveable circles with documents as variously sized rectangles suspended between them
visual and domain tasks

Visualization Type

System

Word

Ordered text such as search engine output

Icon list

Tile Bars, Cougar

Graph (Cartesian)

GUIDO, BIRD, InfoCrystal, Component

State

Spring (physical analogue)

VIBE

VISUAL AND DOMAIN TASKS
  • Basic Forms
    • Map Systems
    • Dimensions & Reference Point Systems
  • Visualization Types
task classification of wehrend lewis
Task Classification of WEHREND & LEWIS
  • Locate
  • Identify
  • Distinguish
  • Categorize
  • Cluster
  • Distribution
  • Rank
  • Compare between relations
  • Associate
  • Correlate
zhou feiner visual task taxonomy

Implication

Type

Subtype

Elemental tasks

Organization

Visual grouping

Proximity

associate, cluster, locate

Similarity

categorize, cluster, distinguish

Continuity

associate, locate, reveal

Closure

cluster, locate, outline

Visual attention

cluster, distinguish, emphasize, locate

Visual sequence

emphasize, identify, rank

Visual composition

associate, correlate, identify, reveal

Signaling

Structuring

tabulate, plot, structure, trace, map

Encoding

label, symbolize, portray, quantify

Transformation

Modification

emphasize, generalize, reveal

Transition

switch

ZHOU & FEINER Visual Task Taxonomy
methodology
METHODOLOGY
  • Dependent Variables
    • Number of correct answers.
    • Time to completion of a task set.
  • Independent Variables
    • Display Type
    • Order of Presentation
    • Individual Task
    • Scenario Difficulty
  • 195 subjects undertook the study using web
  • 2 term or 3 term test randomly
generating experimental tasks

Visual displays

Specific IR subtasks

Taxonomic

categories

Generalized

questions

Generating Experimental Tasks
  • Sample as broadly as possible rather than deeply
  • Select tasks whose parameter lists varied significantly
2 term test
2-Term Test

2.1 Are there more documents that contain ONLY the term Romania or ONLY the term Czechoslovakia?

2.2 Which is the most frequent key term in this set of documents? A. Oil; B. York

2.3 One of the documents is unlike any of the others. Can you identify it? Place the document number in the text box.

2.4 Rank documents A, B, and C with respect to the amount of term ‘Soviet’ that they contain

2.5 Which of the following documents are most similar with respect to the relative amount of the key terms?

2.6 What of the following statements is true?

A. There are no documents that contain roughly equal amounts for the two terms.

B.If a document talks about ‘Oil’ then it also talks about ‘Texas’.

C.‘Texas’ and ‘Oil’ are not very highly related.

D.A and C

E.All of the above

2.7 Location

3 term test
3-Term Test

3-1.Are there more documents that contain ONLY the term ‘earthquake’ or ONLY the term ‘California’ or ONLY the term ‘death’?

3-2.Which is the most frequent key term in this set of documents? A. Vatican; B. Embassy; C. Noriega

3-3.One of the documents is unlike any of the others. Can you identify it? Place the document number in the text box.

3-4.Rank documents A, B, and C with respect to the amount of term ‘Company’ that they contain.

3-5.Which of the following documents are most similar with respect to the relative amount of the key terms?

3-6.Which of the following statements is true?

A. At least one document contains all three terms.

B. At least one document contains the terms ‘Arab’ and ‘bomb’.

C. ‘Vatican’ and ‘Arab’ are not very highly related.

D. B and C

E. All of the above.

3-7.Location

evaluating visualizations using a taxonomic guide15
Evaluating visualizations: using a taxonomic guide

Results – subjects

1. Subjects

No significant differences between the studies for any of these variables.

Mean age in the 2- and 3-term studies was 23.2 and 23.6 years.

The results show that the skill level of subjects of the 2- and 3-term groups were no significant differences.

1.Gender

2.Current educational level

3.Native language

  • No significant differences between the studies for any of these variables.
  • The results show that the skill level of subjects of the 2- and 3-term groups were no significant differences.
evaluating visualizations using a taxonomic guide16
Evaluating visualizations: using a taxonomic guide

Results – time to completion

2. For the 2-term study

  • Significant differences among the display types with respect to completion time (p<0.001).
evaluating visualizations using a taxonomic guide17
Evaluating visualizations: using a taxonomic guide

Results – time to completion

2. For the 2-term study

  • Significant differences among the display types with respect to completion time (p<0.001).
  • Using “spring” as pivot case, all of the other display types are shown to take a significantly longer time in order to complete the task.
evaluating visualizations using a taxonomic guide18
Evaluating visualizations: using a taxonomic guide

Results – time to completion

2. For the 3-term study

  • The ANOVA shows that the four displays were significantly different (p<0.001).
  • Using “spring” as the pivot case, the completion time is highly different from each of the other displays.

Within-subjects contrasts for 3-term display

evaluating visualizations using a taxonomic guide19
Evaluating visualizations: using a taxonomic guide

Results – time to completion

2. Analysis by pair-wise contrasts

  • The “word” and “table” displays were roughly equivalent in terms of speed of performance.
  • The “icon” display was faster.
  • The “spring” display was fastest.
evaluating visualizations using a taxonomic guide20
Evaluating visualizations: using a taxonomic guide

Results – time to completion

2. Comparison across study types (2- and 3-term)

  • Between-subjects factor.

Effect of display type on time to complete task set: 2-term vs. 3-term

evaluating visualizations using a taxonomic guide21
Evaluating visualizations: using a taxonomic guide

Results – time to completion

2. Comparison across study types (2- and 3-term)

  • Between-subjects factor.
  • For the “word”, “icon”, and “table” displays, the subjects required more time in the 3-term conditions in order to complete the tasks than the corresponding 2-term conditions.

Effect of display type on time to complete task set: 2-term vs. 3-term

evaluating visualizations using a taxonomic guide22
Evaluating visualizations: using a taxonomic guide

Results – time to completion

2. Comparison across study types (2- and 3-term)

  • Between-subjects factor.
  • For the “word”, “icon”, and “table” displays, the subjects required more time in the 3-term conditions in order to complete the tasks than the corresponding 2-term conditions.
  • The “spring” display did not achieve significance (p=0.086).

Effect of display type on time to complete task set: 2-term vs. 3-term

evaluating visualizations using a taxonomic guide23
Evaluating visualizations: using a taxonomic guide

Results – correctness of answers

3. Correctness of answers

  • Second method of assessing performance.
evaluating visualizations using a taxonomic guide24
Evaluating visualizations: using a taxonomic guide

Results – correctness of answers

3. Correctness of answers

  • Second method of assessing performance.
  • “Word” display shows a lower number of correct answers than the other displays (pair-wise comparisons all p<0.001).
evaluating visualizations using a taxonomic guide25
Evaluating visualizations: using a taxonomic guide

Results – correctness of answers

3. Correctness of answers

  • Second method of assessing performance.
  • “Word” display shows a lower number of correct answers than the other displays (pair-wise comparisons all p<0.001).
  • No significant differences in number of correct answers between the 2-term and 3-term studies.
evaluating visualizations using a taxonomic guide26
Evaluating visualizations: using a taxonomic guide

Results – order of presentation

4. Order effect (time performance)

  • The order of presentation of the display type was randomized.
  • Poorer performance when the display was presented first in the series.
  • Progressive decreases in the time of the subsequent trials.

2-term study

3-term study

evaluating visualizations using a taxonomic guide27
Evaluating visualizations: using a taxonomic guide

Results – order of presentation

4. Order effect (correctness of answers)

evaluating visualizations using a taxonomic guide28

Order effect

Number of correct answers

Evaluating visualizations: using a taxonomic guide

Results – order of presentation

4. Order effect (correctness of answers)

  • There was no significant effect of the presentation order on performance as measured by the correctness of answers.
evaluating visualizations using a taxonomic guide29
Evaluating visualizations: using a taxonomic guide

5. Performance with respect to task types

  • “Associate”, “identify” and “rank” task

… were performed in very short time periods and

… associated with a very high fraction of correct

answers.

Results – performance with respect to task types

evaluating visualizations using a taxonomic guide30
Evaluating visualizations: using a taxonomic guide

5. Performance with respect to task types

  • “Associate”, “identify” and “rank” task

… were performed in very short time periods and

… associated with a very high fraction of correct

answers.

  • “Cluster”, “locate”, and some of the compare tasks

… were took significantly longer to perform and

… have high fraction of error.

Results – performance with respect to task types

evaluating visualizations using a taxonomic guide31
Evaluating visualizations: using a taxonomic guide

Results – preferences

Results – preferences

6. Preferences (for both the 2- and 3-term studies)

evaluating visualizations using a taxonomic guide32
Evaluating visualizations: using a taxonomic guide

Results – preferences

Results – preferences

6. Preferences (for both the 2- and 3-term studies)

analysis showed that no relationship between time completion and preferences.

  • analysis showed that no relationship between time completion and preferences.
evaluating visualizations using a taxonomic guide33
Evaluating visualizations: using a taxonomic guide

Results – preferences

Results – preferences

6. Preferences (for both the 2- and 3-term studies)

analysis showed that no relationship between time completion and preferences.

  • However, there was a correlation between correctness and preferences.
  • analysis showed that no relationship between time completion and preferences.
evaluating visualizations using a taxonomic guide34
Evaluating visualizations: using a taxonomic guide

Results – preferences

Results – preferences

6. Preferences (for both the 2- and 3-term studies)

analysis showed that no relationship between time completion and preferences.

However, there was a correlation between correctness and preferences.

  • In the non-parametric analysis,

… no correlation between the position in which any display was seen and any positional ranking assigned by the subjects.

analysis showed that no relationship between time completion and preferences.

  • However, there was a correlation between correctness and preferences.
  • analysis showed that no relationship between time completion and preferences.
evaluating visualizations using a taxonomic guide35
Evaluating visualizations: using a taxonomic guide

Discussion

  • The “word” and “text” displays were always associated with poor time performance. (preliminary studies reported earlier)
  • “Spring” display is superior in producing quick responses.
  • A visual taxonomy promises to be a useful guide for developing visual interfaces in general and IR interfaces in particular.
evaluating visualizations using a taxonomic guide36
Evaluating visualizations: using a taxonomic guide

Conclusion

  • Based on the technique of back-to-basics strategy, the visualization techniques themselves were tested, but not the systems.
  • The studies show that the “spring” and “icon” displays can provide an efficient and effective way to present information.
  • The technique of asking questions could be redesigned in order to improve the Internal validity.

---- END ----

evaluating visualizations using a taxonomic guide39
Evaluating visualizations: using a taxonomic guide

Results – order of presentation

4. Order effect (time performance)

  • Statistical analysis show that the first point was different from the others.

2-term study

3-term study

evaluating visualizations using a taxonomic guide40
Evaluating visualizations: using a taxonomic guide

Results – order of presentation

4. Order effect (time performance)

  • Statistical analysis show that the first point was different from the others.
  • However, the subsequent presentations were not different from each other.

2-term study

3-term study

evaluating visualizations using a taxonomic guide41
Evaluating visualizations: using a taxonomic guide

Results – order of presentation

4. Order effect (time performance)

  • The slopes of the lines are initially steeper.
  • The “spring” display appears to be more flattened than the other curves.

2-term study

3-term study

evaluating visualizations using a taxonomic guide42

Order effect

Number of correct answers

Evaluating visualizations: using a taxonomic guide

Results – order of presentation

4. Order effect (correctness of answers)

  • There was no significant effect of the presentation order on performance as measured by the correctness of answers.
  • “Spring” display is the only display that is not influenced by the increased complexity of the 3-term conditions.
evaluating visualizations using a taxonomic guide43
Evaluating visualizations: using a taxonomic guide

5. Paired contrasts

Results – performance with respect to task types

evaluating visualizations using a taxonomic guide44
Evaluating visualizations: using a taxonomic guide

5. Paired contrasts

Results – performance with respect to task types

  • For paired contrasts, using first question (compare) as the pivot group …
evaluating visualizations using a taxonomic guide45
Evaluating visualizations: using a taxonomic guide

5. Paired contrasts

Results – performance with respect to task types

  • For paired contrasts, using first question (compare) as the pivot group …
  • … both performance measures (completion time and correct answers) showed a significant difference for each pair of values …

Completion time

Correctness

evaluating visualizations using a taxonomic guide46
Evaluating visualizations: using a taxonomic guide

5. Paired contrasts

Results – performance with respect to task types

  • For paired contrasts, using first question (compare) as the pivot group …
  • … both performance measures (completion time and correct answers) showed a significant difference for each pair of values …
  • EXCEPT …

1.for the “Distinguish” question for time and

Completion time

evaluating visualizations using a taxonomic guide47
Evaluating visualizations: using a taxonomic guide

5. Paired contrasts

Results – performance with respect to task types

  • For paired contrasts, using first question (compare) as the pivot group …
  • … both performance measures (completion time and correct answers) showed a significant difference for each pair of values …
  • EXCEPT …

1.for the “Distinguish” question for time and

2.for the “Rank”question for correctness.

Correctness