Differences between observed and latent confidence in rank ordering
Download
1 / 19

Differences Between Observed and Latent Confidence in Rank Ordering - PowerPoint PPT Presentation


  • 51 Views
  • Uploaded on

Differences Between Observed and Latent Confidence in Rank Ordering. Brent J. Miller Mark Steyvers University of California, Irvine. Getting Expert Information. Getting Expert Information. Delphi Method. Iterative Collaboration. Independent Rank-Ordering Tasks.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Differences Between Observed and Latent Confidence in Rank Ordering' - loe


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Differences between observed and latent confidence in rank ordering

Differences Between Observed and Latent Confidence in Rank Ordering

Brent J. Miller

Mark Steyvers

University of California, Irvine



Getting expert information1
Getting Expert Information Ordering

Delphi Method

Iterative Collaboration


Independent rank ordering tasks
Independent Rank-Ordering Tasks Ordering

Order these U.S. Presidents, by term of office:

1st Participant

2nd Participant

3rd Participant

Thomas Jefferson

Thomas Jefferson

Thomas Jefferson

John Adams

John Adams

George Washington

John Adams

James Monroe

James Monroe

James Madison

George Washington

James Monroe

James Madison

George Washington

James Madison


Iterative rank ordering tasks
Iterative Rank-Ordering Tasks Ordering

Order these U.S. Presidents, by term of office:

1st Participant

2nd Participant

3rd Participant

George Washington

Thomas Jefferson

George Washington

James Madison

John Adams

Thomas Jefferson

Thomas Jefferson

James Monroe

James Madison

James Madison

John Adams

James Monroe

James Monroe

George Washington

John Adams


Rank ordering experiment
Rank-Ordering Experiment Ordering

  • 100 Subjects

  • 17 Ranking Questions

    • Chronological Events (Holidays, U.S. Presidents)

    • Physical Measures (Landmass, City Pop., State Locations)

    • Ordinal Information (Ten Amendments, Ten Commandments)

  • 2 Conditions


Rank ordering independent iteratives
Rank Ordering – Independent & OrderingIteratives

17 Questions from University Students

x 4

17 Questions via Mechanical Turk


Thurstonian model for rank ordering
Thurstonian Model for Rank Ordering Ordering

A

B

C

σ1

X13

X11

X12

Participant1

Y1: A < B < C


Thurstonian model for rank ordering1
Thurstonian Model for Rank Ordering Ordering

A

B

C

σ2

X13

X11

X12

Participant1

Y1: A < B < C

X23

X21

X22

Participant2

Y2 : A < C < B


Iterative collaboration
Iterative Collaboration Ordering

Position

33 34 35 36 37 38 39

Participants


Thurstonian model for item representation
Thurstonian Model for Item Representation Ordering

A

B

C

σ11

σ12

σ13

X13

X11

X12

Participant1

Y1: A < B < C


Rank ordering experiment1
Rank-Ordering Experiment Ordering

  • 50 Subjects

  • 8 Ranking Questions

    • Chronological Events (Holidays, U.S. Presidents)

    • Physical Measures (Landmass, City Pop., State Locations)

    • Ordinal Information (Ten Amendments, Ten Commandments)

Independent (8)

Iterative (8)

7 5 6 4 1 8 3 2 7 5 6 4 1 8 3 2


Independent vs iterative aggregate performance
Independent vs. Iterative Aggregate Performance Ordering

Independent

Iterative


Independent subject generative model
Independent Subject Generative Model Ordering

i Items

μi

µi ≡ Ground truth

σ2ij ~ InvGamma(.2,1)

xi ~ Norm(µi,σ2ij)

Yj ~ Rank(xj)

yj

xij

σij

j Subjects


Iterative subject generative model
Iterative Subject Generative Model Ordering

cut

μij

µi ≡ Ground truth

σ2ij ~ InvGamma(.2,1)

Xij-1 ~ Norm(µi,σ2ij-1)

Yj-1 ~ Rank(xj-1)

σi,j-1

σij

yj

yj-1

xi,j-1

xij

i Items

j Subjects



Reduction in iterative error by item confidence
Reduction in Iterative Error by Item Confidence Ordering

Confidence for Items with Error > 4 Tau

No Change


Conclusions
Conclusions Ordering

  • Item-level knowledge may allow partial expertise between participants, allowing for improvement in group aggregation.

  • Subject item-level confidence appears to support the validity of our proposed model.

  • We should be able to use subjects’ confidence as an estimate of underlying item-level expertise.