# Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience - PowerPoint PPT Presentation

1 / 76

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience. Nathan D. Grawe Carleton College Serc.carleton.edu/quirk

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

## Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience

Nathan D. Grawe

Carleton College

Serc.carleton.edu/quirk

With support from the US Department of Education’s Fund for the Improvement of Post-Secondary Education, the National Science Foundation, and the WM Keck Foundation.

### What is QR?

The habit of mind to consider the power and limitations of quantitative evidence in the evaluation, construction, and communication of arguments in public, professional, and personal life.

### What is QR?

Four facets of QR:

1) QR requires a basic skill set

### What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

"The test of numeracy, as of any literacy, is whether a person naturally uses appropriate skills in many different contexts"

-National Council on Education and the Disciplines (2001)

### What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

### What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

“Deploying numbers skillfully is as important to communication as deploying verbs.”

-Max Frankel, The New York Times Magazine

### What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

“Numbers [are] the principal language of public argument.”

-BBC Program More or Less

### What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

4) QR is a habit of mind

“[QR] is not a discipline but a way of thinking….” -Lynn Steen

Achieving Quantitative Literacy

### Who is in the measurement sample?

Registered Voters

Eligible Voters

2006 Midterm Elections

### What’s the effect size?

Consider two statements:

• A) With a p<.001 we can conclude that whites in MN, WI, and IA were more likely to vote for Obama than whites in LA, MS, and AL

• B) The fraction of whites who voted for Obama in MN, WI, and IA was at least 35 percentage points higherthan that in LA, MS, and AL (<15% vs. >50%) –forget the p value!

### What’s the research design (correllational or experimental)?

Fact: Those who work with computers earn 15-20% more than others.

“Thus, computer training may, at least in the short run, be a profitable investment for public and private job training programs.”

### What’s the research design (correllational or experimental)?

Other interesting returns:

Calculator = 12.8%

Telephone = 11.4%

Pencil/Pen = 11.2%

Work while sitting = 10.1%

### Controlling for what?

• About 50% of the differential has to do with different career choices….

• 25% percent involves greater time women spend on care-taking….

• The other 25 percent is due to bias and prejudice….”

### What is QR?

“…sophisticated reasoning with elementary mathematics more than elementary reasoning with sophisticated mathematics.”

-Lynn Steen

Achieving Quantitative Literacy

### Implications for Assessment

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves communication

4) QR is a habit of mind

### Implications for Assessment

“[QR] is largely absent from our current systems of assessment and accountability.”

-NCED (2001)

### Implications for Assessment

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves communication

4) QR is a habit of mind

### Implications for Assessment

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves communication

4) QR is a habit of mind

### Implications for Assessment

Application in context:

“Standardized conditions are decontextualized by design"

-Grant Wiggins

“'Get Real!': Assessing for Quantitative Literacy ”

### Implications for Assessment

Communication:

While traditional assessment tools effectively measure comprehension, the ability to read others’ QR exposition does not guarantee the ability to engage in the creation of QR arguments.

### Implications for Assessment

Habit of mind:

“As in book literacy, evidence of students’ ability to play the messy game of the [QR] discipline depends on seeing whether they can handle tasks without specific cues, prompts, or simplifying scaffolds from the teacher-coach or test designer.”

-Grant Wiggins

“'Get Real!': Assessing for Quantitative Literacy”

### Implications for Assessment

"[QR] requires creativity in assessment, since neither course grades nor test scores provide a reliable surrogate."

-Lynn Steen

Achieving Quantitative Literacy

### Implications for Assessment

"The interdisciplinary and contextual nature of [QR] cries out for a cross cutting approach."

-Lynn Steen

Achieving Quantitative Literacy

“We want to regularly assess student work with numbers and numerical ideas in the field….”

-Grant Wiggins

“'Get Real!': Assessing

for Quantitative Literacy”

### The QuIRK Rubric

QuIRK’s idea:

Measure QR in the “natural” context of papers written in courses across the curriculum—papers written for “authentic” purposes.

### The QuIRK Rubric

Sophomore Writing Portfolio

3-5 papers plus reflective essay written in 2 of the 4 college divisions

-observation

-analysis

-interpretation

-documented sources

-thesis-driven argument

### The QuIRK Rubric

Writing requirement assessment:

30 faculty members, 3 days, 450 portfolios

QR assessment:

6-8 faculty members, 3 days, random sample of portfolios, 1 paper each drawn from analysis, interpretation, or observation categories

### The QuIRK Rubric

1. A little book-keeping

### The QuIRK Rubric

2. Did the student take the paper in a QR-relevant direction?

### The QuIRK Rubric

Central Use:

Use of numbers to address a central question, issue, or theme

Peripheral Use:

Use of numbers to provide useful detail, enrich descriptions, present background, or establish frames of reference

### The QuIRK Rubric

The importance of the periphery:

“Even for works that are not inherently quantitative, one or two numeric facts can help convey the importance or context of your topic.”

-Jane Miller

The Chicago Guide to

### The QuIRK Rubric

3. If relevant, to what extent did the student actually use QR?

### The QuIRK Rubric

4. If present, rate the holistic quality of the QR.

### The QuIRK Rubric

5. Spotlight on several recurring problems:

### The QuIRK Rubric

6. Does assignment explicitly call for QR?

### The QuIRK Rubric

Inter-Rater Agreement Statistics

QR Relevance:75%

QR Extent:82%

QR Quality:67%

Problem Characteristics:68%

### Examples from Student Work

Example of QR-irrelevant paper:

“The Maiden who Needs No Saving”—an analysis of Keat’s treatment of helplessness and power in “La Belle Dame Sans Merci”

### Examples from Student Work

Example of ineffective peripheral QR paper:

Title: Untitled

Thesis: A synthesis of 3 alternative views on poverty provides a more complete picture than any 1 perspective alone

\$192 billion

\$60 billion

1/65th of GNP

1/230th of GNP

### Examples from Student Work

Example of effective peripheral QR paper:

Title: “Les Banlieues d’Islam”

Thesis:

### Examples from Student Work

Example of ineffective centrally QR paper:

Title: Untitled

Thesis: Based on data from a time diary and other sources, the American quality of life is better than that in less-developed societies.

### Examples from Student Work

Example of ineffective centrally QR paper:

“Day Care and Development”

Topic: Literature review of effects of day care participation on child development

### Examples from Student Work

Even though quantitative evidence is referenced extensively, student doesn’t appreciate the real power of the actual numbers:

Ex: “The amount of time that children spent in day-care was positively correlated with reports of how many friends they had.”

### Examples from Student Work

Example of effective centrally QR paper:

Title: “Modeling the Implicit Learning of Language Acquisition”

Thesis: Even if people aren’t aware of the systems undergirding language, they implicitly learn the system through experience.

### Examples from Student Work

“In phase two, the mean number of correct string classifications the subjects gave was 17.3 out of 24; all but two participants correctly identified more than 12 strings, and the mode number of correct responses was 18….”

[Discussion of statistical test.]

“It was expected that the participants would be able to correctly classify more than 12 of the 24 strings in phase two, so the result is not especially surprising. Still, the data provide a striking example of implicit learning. ”

### Infusing QR Across the Curriculum

“…numeracy is not something mastered in a single course….Thus quantitative material needs to permeate the curriculum, … so that students have opportunities to practice their skills and see how useful they can be in understanding a wide range of problems.”

-Derek Bok (2006)

“…authentic and enduring learning…can rarely succeed one course at a time.”

-Lee Shulman (1997)

### Infusing QR Across the Curriculum

“If [QR] remains the responsibility solely of mathematics departments—especially if it is caged into a single course such as ‘Math for Liberal Arts’—students will continue to see [QR] as something that happens only in the mathematics classroom.”

-Lynn Steen

Achieving Quantitative Literacy

### Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

### Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

### Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

### Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

### Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

### Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

### Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

### Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

### Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant

### Infusing QR Across the Curriculum

• 30% of gen ed papers are centrally QR relevant

• Of these, nearly 50% are outside natural sciences

• Another 25% are peripherally so

• Of these, 90+% are outside natural sciences and more than half are in Arts, Lit, & Humanities

### Infusing QR Across the Curriculum

By QR relevance, what fraction of papers use QR little/none, some, and extensively?

### Infusing QR Across the Curriculum

“Weasely” examples:A) “The downtown areas of large cities are often home to copious franchises.”

B) Recall Daycare Paper

### Our Curricular Response

Disciplines Currently Represented:

American Studies

Biology (2)

Chemistry

Economics (5)

Environmental Science

English

Fine Arts

Geography (2)

Geoscience

Mathematics (12)

History (2)

Physics

Political Science (8)

Psychology (4)

Sociology (4)

### Our Curricular Response

Coming Soon!:

Anthropology

Biology

Classics

Economics (2)

English

French (2)

History (3)

Political Science (4)

Psychology

Religion

Sociology (2)

### Our Curricular Response

Example 1: An Ill-Structured Physics Lab

Example 2: Intro to Lintina/o Studies Paper

Example 3: History Paper on India’s Colonial Census

Example 4: Exploring and Architectural Remodel

### Our Curricular Response

Example 5: Environmental History Paper

Example 6: Development Economics Paper

Example 7: Native-American Literature Analysis

### Our Curricular Response

Suggestions for Assignment Creation:

Role

Audience

Format