are all questions created equal factors that influence cloze question difficulty n.
Download
Skip this Video
Download Presentation
Are all questions created equal?: Factors that influence cloze question difficulty.

Loading in 2 Seconds...

play fullscreen
1 / 28

Are all questions created equal?: Factors that influence cloze question difficulty. - PowerPoint PPT Presentation


  • 55 Views
  • Uploaded on

Are all questions created equal?: Factors that influence cloze question difficulty. Brooke Soden Hensler Carnegie Mellon University (starting graduate school at Florida Center for Reading Research this Fall) Joseph E. Beck Carnegie Mellon University.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Are all questions created equal?: Factors that influence cloze question difficulty.' - odysseus-marks


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
are all questions created equal factors that influence cloze question difficulty

Are all questions created equal?:Factors that influence cloze question difficulty.

Brooke Soden Hensler

Carnegie Mellon University

(starting graduate school at

Florida Center for Reading Research this Fall)

Joseph E. BeckCarnegie Mellon University

Society for the Scientific Study of Reading – July 2006

Funding: National Science Foundation

why look at multiple choice cloze questions
Why Look at Multiple Choice Cloze Questions?
  • Multiple Choice Cloze are widely used assessments of comprehension
  • Problem: outcome measure is typically binary (little information about student).
  • Goal: use multiple choice cloze questions to…
    • More accurately assess students
    • Track student reading development
    • Better understand what makes cloze questions hard
project listen s computer reading tutor mostow aist 2001
Project LISTEN’s Computer Reading Tutor(Mostow & Aist, 2001)
  • Automated
  • Students use throughout year
  • Accompanying paper standardized test scores (pre & post)
a question appears reading tutor reads both question and response choices mostow et al 2004
A question appears…*Reading Tutor reads both Question and Response Choices.(Mostow, et al., 2004)
reading tutor advantages
Reading Tutor Advantages
  • Well-specified & unbiased question construction (randomly generated)
  • Questions automatically administered, scored, & recorded
  • Longitudinal collection over school year
  • Large N (students & questions)
how many q s from whom data description
How many Q’s from Whom?Data Description
  • 81,175 Questions
  • 1042 Students
  • 11 = Median number of questions answered
    • (Many students infrequent users of tutor)
  • 2001-02 & 2002-03 School years
  • Diverse population in Pittsburgh area
research questions
Research Questions
  • Is a particular part of speech (e.g., nouns, verbs, etc.) more difficult for students?
    • If nouns are learned first (Gentner, 1982; Golinkoff, et al., 2000), might students be more proficient at answering noun questions?
  • Which factors influence question difficulty?
  • How can we better assess students using multiple choice cloze questions?
    • Vocabulary researchers have given partial credit for correct part of speech (e.g., Schwanenflugel, et al., 1997)
approach
Approach
  • Build logistic regression model to predict individual question performance
    • Terms in model: student identity, part of speech of answer, properties of question (e.g., question length)
  • Advantages of modeling approach
    • Simultaneously estimates impact of question properties and student proficiency on question performance
    • Makes use of all ~80k questions
effect of parts of speech
Effect of Parts of Speech

<

<

<

Nouns

Verbs

Adjectives

Adverbs

(p < 0.001)

(p < 0.05)

(p < 0.001)

effect of parts of speech1
Effect of Parts of Speech

<

<

<

Nouns

Verbs

Adjectives

Adverbs

(p < 0.001)

(p < 0.05)

(p < 0.001)

harder

easier

impact of other part of speech terms
Impact of other Part of Speech terms

Difficulty Significance

Most Common  p < 0.01

Part of Speech

# of Choices  p < 0.001

with Answer’s POS

“Sally had to _______ her lips when she heard the news.”

(cloud, purse, holds, magnificent)

“Henry read his _______ under the tree.”

(cup, dog, book, hair)

impact of other part of speech terms1
Impact of other Part of Speech terms

Difficulty Significance

Most Commonp < 0.01

Part of Speech

# of Choices  p < 0.001

with Answer’s POS

“Henry read his _______ under the tree.”

(cup, dog,book, hair)

“Sally had to _______ her lips when she heard the news.”

(lamp,purse, beautiful, magnificent)

 more common POS = easier

 less common POS = harder

impact of other part of speech terms2
Impact of other Part of Speech terms

Difficulty Significance

Most Common  p < 0.01

Part of Speech

# of Choicesp < 0.001

with Answer’s POS

“Henry read his _______ under the tree.”

(cup, dog, book, hair)

“Sally had to _______ her lips when she heard the news.”

(lamp, purse, beautiful, magnificent)

(noun)

 more choices with correct POS = harder

(verb)

  • fewer choices

with correct POS

= easier

impact of other terms
Impact of other terms

Difficulty Significance

Question  p < 0.001

Length

Deletion  p < 0.001

Location

“We can _______ the stars in the sky despite the bright city lights around us.”

(at, with, most, see)

“They rode their _______ .”

(farmer, bikes, play, blue)

impact of other terms1
Impact of other terms

Difficulty Significance

Questionp < 0.001

Length

Deletion  p < 0.001

Location

“We can _______ the stars in the sky despite the bright city lights around us.”

(at, with, most, see)

“They rode their _______ .”

(farmer, bikes, play, blue)

 longer = harder

 shorter = easier

impact of other terms2
Impact of other terms

Difficulty Significance

Question  p < 0.001

Length

Deletionp < 0.001

Location

“We can _______ the stars in the sky despite the bright city lights around us.”

(at, with, most, see)

“They rode their _______ .”

(farmer, bikes, play, blue)

 blank earlier = harder

 blank later = easier

using model to assess student reading comprehension
Using model to assess student reading comprehension
  • Model estimates Beta parameter for each student
    • Represents how well student did at answering cloze questions (controlling for difficulty factors)
    • Should correlate with external comprehension measure
  • Compare Beta vs. percent correct for predicting WRMT comprehension composite*
    • Student Beta: r = .644, p < .001
    • Percent correct: r = .507, p < .001
    • Reliability of difference in correlations, p < .01
  • Also provides check on validity of regression model

*N = 465, 1 extreme outlier was eliminated from analyses.

conclusions
Conclusions
  • Length of question, location of deleted word, and part of speech of correct answer affect question difficulty.
  • Logistic regression is a strong choice for analyzing cloze data.
  • Multiple-choice cloze questions can assess a student at a more accurate level than current practice.
questions
Questions?
  • Nominated for Best Paper Award:

Soden Hensler, B., Beck, J. E. (2006). Better student assessing by

finding difficulty factors in a fully automated comprehension

measure. Intelligent Tutoring Systems.

  • Brooke Soden Hensler

bsodenhensler@gmail.com

  • Joseph E. Beck

joseph.beck@gmail.com

  • Project LISTEN & The Reading Tutor

http://www.cs.cmu.edu/~listen/

references
References
  • Gentner, D. (1981). Some interesting differences between verbs and nouns.Cognition and Brain Theory, 4(2).
  • Golinkoff, R.M., Hirsh-Pasek, K., Bloom, L., Smith, L. B., Woodward, A. L., Akhtar, N., Tomasello, M., & Hollich, G. (2000). Becoming a word learner: A debate on lexical acquisition. New York: Oxford University Press.
  • Mostow, J. & Aist, G. (2001). Evaluating tutors that listen: An overview of Project LISTEN. In K. Forbus & P. Feltovich (Eds.), Smart Machines in Education (169 - 234) Menlo Park, CA: MIT/AAAI Press.
  • Mostow, J., Beck, J. E., Bey, J., Cuneo, A., Sison, J., Tobin, B. & Valeri, J. (2004). Using automated questions to assess reading comprehension, vocabulary, and effects of tutorial interventions.Technology, Instruction, Cognition and Learning, 2, p. 97-134
  • Schwanenflugel, P.J., Stahl, S. A., & McFalls, E. L. (1997). Partial word knowledge and vocabulary growth during reading comprehension. Journal of Literacy Research, 29(4).
developmental trends in learning parts of speech1
Developmental Trends in Learning Parts of Speech

p = .52

p = .64

p = .99

p = .71

p < .001

syntactic awareness
Syntactic Awareness

p = .73

p = .48

p = .01

p = .02

p < .001

slide28
Effect of Part of Speech*Interpretation: positive Beta means student is more likely to answer question correctly