differences between observed and latent confidence in rank ordering
Download
Skip this Video
Download Presentation
Differences Between Observed and Latent Confidence in Rank Ordering

Loading in 2 Seconds...

play fullscreen
1 / 19

Differences Between Observed and Latent Confidence in Rank Ordering - PowerPoint PPT Presentation


  • 51 Views
  • Uploaded on

Differences Between Observed and Latent Confidence in Rank Ordering. Brent J. Miller Mark Steyvers University of California, Irvine. Getting Expert Information. Getting Expert Information. Delphi Method. Iterative Collaboration. Independent Rank-Ordering Tasks.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Differences Between Observed and Latent Confidence in Rank Ordering' - loe


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
differences between observed and latent confidence in rank ordering

Differences Between Observed and Latent Confidence in Rank Ordering

Brent J. Miller

Mark Steyvers

University of California, Irvine

getting expert information1
Getting Expert Information

Delphi Method

Iterative Collaboration

independent rank ordering tasks
Independent Rank-Ordering Tasks

Order these U.S. Presidents, by term of office:

1st Participant

2nd Participant

3rd Participant

Thomas Jefferson

Thomas Jefferson

Thomas Jefferson

John Adams

John Adams

George Washington

John Adams

James Monroe

James Monroe

James Madison

George Washington

James Monroe

James Madison

George Washington

James Madison

iterative rank ordering tasks
Iterative Rank-Ordering Tasks

Order these U.S. Presidents, by term of office:

1st Participant

2nd Participant

3rd Participant

George Washington

Thomas Jefferson

George Washington

James Madison

John Adams

Thomas Jefferson

Thomas Jefferson

James Monroe

James Madison

James Madison

John Adams

James Monroe

James Monroe

George Washington

John Adams

rank ordering experiment
Rank-Ordering Experiment
  • 100 Subjects
  • 17 Ranking Questions
    • Chronological Events (Holidays, U.S. Presidents)
    • Physical Measures (Landmass, City Pop., State Locations)
    • Ordinal Information (Ten Amendments, Ten Commandments)
  • 2 Conditions
rank ordering independent iteratives
Rank Ordering – Independent & Iteratives

17 Questions from University Students

x 4

17 Questions via Mechanical Turk

thurstonian model for rank ordering
Thurstonian Model for Rank Ordering

A

B

C

σ1

X13

X11

X12

Participant1

Y1: A < B < C

thurstonian model for rank ordering1
Thurstonian Model for Rank Ordering

A

B

C

σ2

X13

X11

X12

Participant1

Y1: A < B < C

X23

X21

X22

Participant2

Y2 : A < C < B

iterative collaboration
Iterative Collaboration

Position

33 34 35 36 37 38 39

Participants

thurstonian model for item representation
Thurstonian Model for Item Representation

A

B

C

σ11

σ12

σ13

X13

X11

X12

Participant1

Y1: A < B < C

rank ordering experiment1
Rank-Ordering Experiment
  • 50 Subjects
  • 8 Ranking Questions
    • Chronological Events (Holidays, U.S. Presidents)
    • Physical Measures (Landmass, City Pop., State Locations)
    • Ordinal Information (Ten Amendments, Ten Commandments)

Independent (8)

Iterative (8)

7 5 6 4 1 8 3 2 7 5 6 4 1 8 3 2

independent subject generative model
Independent Subject Generative Model

i Items

μi

µi ≡ Ground truth

σ2ij ~ InvGamma(.2,1)

xi ~ Norm(µi,σ2ij)

Yj ~ Rank(xj)

yj

xij

σij

j Subjects

iterative subject generative model
Iterative Subject Generative Model

cut

μij

µi ≡ Ground truth

σ2ij ~ InvGamma(.2,1)

Xij-1 ~ Norm(µi,σ2ij-1)

Yj-1 ~ Rank(xj-1)

σi,j-1

σij

yj

yj-1

xi,j-1

xij

i Items

j Subjects

reduction in iterative error by item confidence
Reduction in Iterative Error by Item Confidence

Confidence for Items with Error > 4 Tau

No Change

conclusions
Conclusions
  • Item-level knowledge may allow partial expertise between participants, allowing for improvement in group aggregation.
  • Subject item-level confidence appears to support the validity of our proposed model.
  • We should be able to use subjects’ confidence as an estimate of underlying item-level expertise.
ad