assessing quantitative reasoning in student writing a quirky experience
Download
Skip this Video
Download Presentation
Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience

Loading in 2 Seconds...

play fullscreen
1 / 76

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience - PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience. Nathan D. Grawe Carleton College Serc.carleton.edu/quirk

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience' - braima


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
assessing quantitative reasoning in student writing a quirky experience

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience

Nathan D. Grawe

Carleton College

Serc.carleton.edu/quirk

With support from the US Department of Education’s Fund for the Improvement of Post-Secondary Education, the National Science Foundation, and the WM Keck Foundation.

what is qr1
What is QR?

The habit of mind to consider the power and limitations of quantitative evidence in the evaluation, construction, and communication of arguments in public, professional, and personal life.

what is qr2
What is QR?

Four facets of QR:

1) QR requires a basic skill set

what is qr3
What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

"The test of numeracy, as of any literacy, is whether a person naturally uses appropriate skills in many different contexts"

-National Council on Education and the Disciplines (2001)

what is qr4
What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

what is qr5
What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

“Deploying numbers skillfully is as important to communication as deploying verbs.”

-Max Frankel, The New York Times Magazine

what is qr6
What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

“Numbers [are] the principal language of public argument.”

-BBC Program More or Less

what is qr7
What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

4) QR is a habit of mind

“[QR] is not a discipline but a way of thinking….” -Lynn Steen

Achieving Quantitative Literacy

who is in the measurement sample
Who is in the measurement sample?

Registered Voters

Eligible Voters

2006 Midterm Elections

what s the effect size1
What’s the effect size?

Consider two statements:

  • A) With a p<.001 we can conclude that whites in MN, WI, and IA were more likely to vote for Obama than whites in LA, MS, and AL
  • B) The fraction of whites who voted for Obama in MN, WI, and IA was at least 35 percentage points higherthan that in LA, MS, and AL (<15% vs. >50%) –forget the p value!
what s the research design correllational or experimental
What’s the research design (correllational or experimental)?

Fact: Those who work with computers earn 15-20% more than others.

“Thus, computer training may, at least in the short run, be a profitable investment for public and private job training programs.”

what s the research design correllational or experimental1
What’s the research design (correllational or experimental)?

Other interesting returns:

Calculator = 12.8%

Telephone = 11.4%

Pencil/Pen = 11.2%

Work while sitting = 10.1%

controlling for what1
Controlling for what?
  • About 50% of the differential has to do with different career choices….
  • 25% percent involves greater time women spend on care-taking….
  • The other 25 percent is due to bias and prejudice….”
what is qr8
What is QR?

“…sophisticated reasoning with elementary mathematics more than elementary reasoning with sophisticated mathematics.”

-Lynn Steen

Achieving Quantitative Literacy

implications for assessment
Implications for Assessment

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves communication

4) QR is a habit of mind

implications for assessment1
Implications for Assessment

“[QR] is largely absent from our current systems of assessment and accountability.”

-NCED (2001)

implications for assessment2
Implications for Assessment

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves communication

4) QR is a habit of mind

implications for assessment3
Implications for Assessment

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves communication

4) QR is a habit of mind

implications for assessment4
Implications for Assessment

Application in context:

“Standardized conditions are decontextualized by design"

-Grant Wiggins

“'Get Real!': Assessing for Quantitative Literacy ”

implications for assessment5
Implications for Assessment

Communication:

While traditional assessment tools effectively measure comprehension, the ability to read others’ QR exposition does not guarantee the ability to engage in the creation of QR arguments.

implications for assessment6
Implications for Assessment

Habit of mind:

“As in book literacy, evidence of students’ ability to play the messy game of the [QR] discipline depends on seeing whether they can handle tasks without specific cues, prompts, or simplifying scaffolds from the teacher-coach or test designer.”

-Grant Wiggins

“'Get Real!': Assessing for Quantitative Literacy”

implications for assessment7
Implications for Assessment

"[QR] requires creativity in assessment, since neither course grades nor test scores provide a reliable surrogate."

-Lynn Steen

Achieving Quantitative Literacy

implications for assessment8
Implications for Assessment

"The interdisciplinary and contextual nature of [QR] cries out for a cross cutting approach."

-Lynn Steen

Achieving Quantitative Literacy

“We want to regularly assess student work with numbers and numerical ideas in the field….”

-Grant Wiggins

“'Get Real!': Assessing

for Quantitative Literacy”

the quirk rubric
The QuIRK Rubric

QuIRK’s idea:

Measure QR in the “natural” context of papers written in courses across the curriculum—papers written for “authentic” purposes.

the quirk rubric1
The QuIRK Rubric

Sophomore Writing Portfolio

3-5 papers plus reflective essay written in 2 of the 4 college divisions

-observation

-analysis

-interpretation

-documented sources

-thesis-driven argument

the quirk rubric2
The QuIRK Rubric

Writing requirement assessment:

30 faculty members, 3 days, 450 portfolios

QR assessment:

6-8 faculty members, 3 days, random sample of portfolios, 1 paper each drawn from analysis, interpretation, or observation categories

the quirk rubric3
The QuIRK Rubric

1. A little book-keeping

the quirk rubric4
The QuIRK Rubric

2. Did the student take the paper in a QR-relevant direction?

the quirk rubric5
The QuIRK Rubric

Central Use:

Use of numbers to address a central question, issue, or theme

Peripheral Use:

Use of numbers to provide useful detail, enrich descriptions, present background, or establish frames of reference

the quirk rubric6
The QuIRK Rubric

The importance of the periphery:

“Even for works that are not inherently quantitative, one or two numeric facts can help convey the importance or context of your topic.”

-Jane Miller

The Chicago Guide to

Writing About Numbers

the quirk rubric7
The QuIRK Rubric

3. If relevant, to what extent did the student actually use QR?

the quirk rubric8
The QuIRK Rubric

4. If present, rate the holistic quality of the QR.

the quirk rubric9
The QuIRK Rubric

5. Spotlight on several recurring problems:

the quirk rubric10
The QuIRK Rubric

6. Does assignment explicitly call for QR?

the quirk rubric11
The QuIRK Rubric

Inter-Rater Agreement Statistics

QR Relevance: 75%

QR Extent: 82%

QR Quality: 67%

Problem Characteristics: 68%

examples from student work
Examples from Student Work

Example of QR-irrelevant paper:

“The Maiden who Needs No Saving”—an analysis of Keat’s treatment of helplessness and power in “La Belle Dame Sans Merci”

examples from student work1
Examples from Student Work

Example of ineffective peripheral QR paper:

Title: Untitled

Thesis: A synthesis of 3 alternative views on poverty provides a more complete picture than any 1 perspective alone

examples from student writing
Examples from Student Writing

$192 billion

$60 billion

1/65th of GNP

1/230th of GNP

examples from student work2
Examples from Student Work

Example of effective peripheral QR paper:

Title: “Les Banlieues d’Islam”

Thesis:

examples from student work3
Examples from Student Work

Example of ineffective centrally QR paper:

Title: Untitled

Thesis: Based on data from a time diary and other sources, the American quality of life is better than that in less-developed societies.

examples from student work4
Examples from Student Work

Example of ineffective centrally QR paper:

“Day Care and Development”

Topic: Literature review of effects of day care participation on child development

examples from student work5
Examples from Student Work

Even though quantitative evidence is referenced extensively, student doesn’t appreciate the real power of the actual numbers:

Ex: “The amount of time that children spent in day-care was positively correlated with reports of how many friends they had.”

examples from student work6
Examples from Student Work

Example of effective centrally QR paper:

Title: “Modeling the Implicit Learning of Language Acquisition”

Thesis: Even if people aren’t aware of the systems undergirding language, they implicitly learn the system through experience.

examples from student work7
Examples from Student Work

“In phase two, the mean number of correct string classifications the subjects gave was 17.3 out of 24; all but two participants correctly identified more than 12 strings, and the mode number of correct responses was 18….”

[Discussion of statistical test.]

“It was expected that the participants would be able to correctly classify more than 12 of the 24 strings in phase two, so the result is not especially surprising. Still, the data provide a striking example of implicit learning. ”

infusing qr across the curriculum
Infusing QR Across the Curriculum

“…numeracy is not something mastered in a single course….Thus quantitative material needs to permeate the curriculum, … so that students have opportunities to practice their skills and see how useful they can be in understanding a wide range of problems.”

-Derek Bok (2006)

“…authentic and enduring learning…can rarely succeed one course at a time.”

-Lee Shulman (1997)

infusing qr across the curriculum1
Infusing QR Across the Curriculum

“If [QR] remains the responsibility solely of mathematics departments—especially if it is caged into a single course such as ‘Math for Liberal Arts’—students will continue to see [QR] as something that happens only in the mathematics classroom.”

-Lynn Steen

Achieving Quantitative Literacy

infusing qr across the curriculum2
Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

infusing qr across the curriculum3
Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

infusing qr across the curriculum4
Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

infusing qr across the curriculum5
Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

infusing qr across the curriculum6
Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

infusing qr across the curriculum7
Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

infusing qr across the curriculum8
Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

infusing qr across the curriculum9
Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

infusing qr across the curriculum10
Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

infusing qr across the curriculum11
Infusing QR Across the Curriculum
  • 30% of gen ed papers are centrally QR relevant
    • Of these, nearly 50% are outside natural sciences
  • Another 25% are peripherally so
    • Of these, 90+% are outside natural sciences and more than half are in Arts, Lit, & Humanities
infusing qr across the curriculum12
Infusing QR Across the Curriculum

By QR relevance, what fraction of papers use QR little/none, some, and extensively?

infusing qr across the curriculum13
Infusing QR Across the Curriculum

“Weasely” examples:A) “The downtown areas of large cities are often home to copious franchises.”

B) Recall Daycare Paper

our curricular response4
Our Curricular Response

Disciplines Currently Represented:

American Studies

Biology (2)

Chemistry

Economics (5)

Environmental Science

English

Fine Arts

Geography (2)

Geoscience

Mathematics (12)

History (2)

Physics

Political Science (8)

Psychology (4)

Sociology (4)

our curricular response5
Our Curricular Response

Coming Soon!:

Anthropology

Biology

Classics

Economics (2)

English

French (2)

History (3)

Political Science (4)

Psychology

Religion

Sociology (2)

our curricular response6
Our Curricular Response

Example 1: An Ill-Structured Physics Lab

Example 2: Intro to Lintina/o Studies Paper

Example 3: History Paper on India’s Colonial Census

Example 4: Exploring and Architectural Remodel

our curricular response7
Our Curricular Response

Example 5: Environmental History Paper

Example 6: Development Economics Paper

Example 7: Native-American Literature Analysis

our curricular response8
Our Curricular Response

Suggestions for Assignment Creation:

Role

Audience

Format

Task

Task as

Ill-Structured

Problem

ad