Assessing quantitative reasoning in student writing a quirky experience
This presentation is the property of its rightful owner.
Sponsored Links
1 / 76

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on
  • Presentation posted in: General

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience. Nathan D. Grawe Carleton College Serc.carleton.edu/quirk

Download Presentation

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Assessing quantitative reasoning in student writing a quirky experience

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience

Nathan D. Grawe

Carleton College

Serc.carleton.edu/quirk

With support from the US Department of Education’s Fund for the Improvement of Post-Secondary Education, the National Science Foundation, and the WM Keck Foundation.


What is qr

What is QR?


What is qr1

What is QR?

The habit of mind to consider the power and limitations of quantitative evidence in the evaluation, construction, and communication of arguments in public, professional, and personal life.


What is qr2

What is QR?

Four facets of QR:

1) QR requires a basic skill set


What is qr3

What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

"The test of numeracy, as of any literacy, is whether a person naturally uses appropriate skills in many different contexts"

-National Council on Education and the Disciplines (2001)


What is qr4

What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument


What is qr5

What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

“Deploying numbers skillfully is as important to communication as deploying verbs.”

-Max Frankel, The New York Times Magazine


What is qr6

What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

“Numbers [are] the principal language of public argument.”

-BBC Program More or Less


What is qr7

What is QR?

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves argument

4) QR is a habit of mind

“[QR] is not a discipline but a way of thinking….” -Lynn Steen

Achieving Quantitative Literacy


What do the numbers show

What do the numbers show?


What do the numbers show1

What do the numbers show?


How representative is that

How representative is that?


Compared to what

Compared to what?


How was the variable operationalized

How was the variable operationalized?


Who is in the measurement sample

Who is in the measurement sample?

Registered Voters

Eligible Voters

2006 Midterm Elections


Is the outcome statistically significant

Is the outcome statistically significant?


What s the effect size

What’s the effect size?


What s the effect size1

What’s the effect size?

Consider two statements:

  • A) With a p<.001 we can conclude that whites in MN, WI, and IA were more likely to vote for Obama than whites in LA, MS, and AL

  • B) The fraction of whites who voted for Obama in MN, WI, and IA was at least 35 percentage points higherthan that in LA, MS, and AL (<15% vs. >50%) –forget the p value!


What s the research design correllational or experimental

What’s the research design (correllational or experimental)?

Fact: Those who work with computers earn 15-20% more than others.

“Thus, computer training may, at least in the short run, be a profitable investment for public and private job training programs.”


What s the research design correllational or experimental1

What’s the research design (correllational or experimental)?

Other interesting returns:

Calculator = 12.8%

Telephone = 11.4%

Pencil/Pen = 11.2%

Work while sitting = 10.1%


Controlling for what

Controlling for what?


Controlling for what1

Controlling for what?

  • About 50% of the differential has to do with different career choices….

  • 25% percent involves greater time women spend on care-taking….

  • The other 25 percent is due to bias and prejudice….”


What is qr8

What is QR?

“…sophisticated reasoning with elementary mathematics more than elementary reasoning with sophisticated mathematics.”

-Lynn Steen

Achieving Quantitative Literacy


Implications for assessment

Implications for Assessment

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves communication

4) QR is a habit of mind


Implications for assessment1

Implications for Assessment

“[QR] is largely absent from our current systems of assessment and accountability.”

-NCED (2001)


Implications for assessment2

Implications for Assessment

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves communication

4) QR is a habit of mind


Implications for assessment3

Implications for Assessment

Four facets of QR:

1) QR requires a basic skill set

2) QR demands application in context

3) QR involves communication

4) QR is a habit of mind


Implications for assessment4

Implications for Assessment

Application in context:

“Standardized conditions are decontextualized by design"

-Grant Wiggins

“'Get Real!': Assessing for Quantitative Literacy ”


Implications for assessment5

Implications for Assessment

Communication:

While traditional assessment tools effectively measure comprehension, the ability to read others’ QR exposition does not guarantee the ability to engage in the creation of QR arguments.


Implications for assessment6

Implications for Assessment

Habit of mind:

“As in book literacy, evidence of students’ ability to play the messy game of the [QR] discipline depends on seeing whether they can handle tasks without specific cues, prompts, or simplifying scaffolds from the teacher-coach or test designer.”

-Grant Wiggins

“'Get Real!': Assessing for Quantitative Literacy”


Implications for assessment7

Implications for Assessment

"[QR] requires creativity in assessment, since neither course grades nor test scores provide a reliable surrogate."

-Lynn Steen

Achieving Quantitative Literacy


Implications for assessment8

Implications for Assessment

"The interdisciplinary and contextual nature of [QR] cries out for a cross cutting approach."

-Lynn Steen

Achieving Quantitative Literacy

“We want to regularly assess student work with numbers and numerical ideas in the field….”

-Grant Wiggins

“'Get Real!': Assessing

for Quantitative Literacy”


The quirk rubric

The QuIRK Rubric

QuIRK’s idea:

Measure QR in the “natural” context of papers written in courses across the curriculum—papers written for “authentic” purposes.


The quirk rubric1

The QuIRK Rubric

Sophomore Writing Portfolio

3-5 papers plus reflective essay written in 2 of the 4 college divisions

-observation

-analysis

-interpretation

-documented sources

-thesis-driven argument


The quirk rubric2

The QuIRK Rubric

Writing requirement assessment:

30 faculty members, 3 days, 450 portfolios

QR assessment:

6-8 faculty members, 3 days, random sample of portfolios, 1 paper each drawn from analysis, interpretation, or observation categories


The quirk rubric3

The QuIRK Rubric

1. A little book-keeping


The quirk rubric4

The QuIRK Rubric

2. Did the student take the paper in a QR-relevant direction?


The quirk rubric5

The QuIRK Rubric

Central Use:

Use of numbers to address a central question, issue, or theme

Peripheral Use:

Use of numbers to provide useful detail, enrich descriptions, present background, or establish frames of reference


The quirk rubric6

The QuIRK Rubric

The importance of the periphery:

“Even for works that are not inherently quantitative, one or two numeric facts can help convey the importance or context of your topic.”

-Jane Miller

The Chicago Guide to

Writing About Numbers


The quirk rubric7

The QuIRK Rubric

3. If relevant, to what extent did the student actually use QR?


The quirk rubric8

The QuIRK Rubric

4. If present, rate the holistic quality of the QR.


The quirk rubric9

The QuIRK Rubric

5. Spotlight on several recurring problems:


The quirk rubric10

The QuIRK Rubric

6. Does assignment explicitly call for QR?


The quirk rubric11

The QuIRK Rubric

Inter-Rater Agreement Statistics

QR Relevance:75%

QR Extent:82%

QR Quality:67%

Problem Characteristics:68%


Examples from student work

Examples from Student Work

Example of QR-irrelevant paper:

“The Maiden who Needs No Saving”—an analysis of Keat’s treatment of helplessness and power in “La Belle Dame Sans Merci”


Examples from student work1

Examples from Student Work

Example of ineffective peripheral QR paper:

Title: Untitled

Thesis: A synthesis of 3 alternative views on poverty provides a more complete picture than any 1 perspective alone


Examples from student writing

Examples from Student Writing

$192 billion

$60 billion

1/65th of GNP

1/230th of GNP


Examples from student work2

Examples from Student Work

Example of effective peripheral QR paper:

Title: “Les Banlieues d’Islam”

Thesis:


Examples from student work3

Examples from Student Work

Example of ineffective centrally QR paper:

Title: Untitled

Thesis: Based on data from a time diary and other sources, the American quality of life is better than that in less-developed societies.


Examples from student work4

Examples from Student Work

Example of ineffective centrally QR paper:

“Day Care and Development”

Topic: Literature review of effects of day care participation on child development


Examples from student work5

Examples from Student Work

Even though quantitative evidence is referenced extensively, student doesn’t appreciate the real power of the actual numbers:

Ex: “The amount of time that children spent in day-care was positively correlated with reports of how many friends they had.”


Examples from student work6

Examples from Student Work

Example of effective centrally QR paper:

Title: “Modeling the Implicit Learning of Language Acquisition”

Thesis: Even if people aren’t aware of the systems undergirding language, they implicitly learn the system through experience.


Examples from student work7

Examples from Student Work

“In phase two, the mean number of correct string classifications the subjects gave was 17.3 out of 24; all but two participants correctly identified more than 12 strings, and the mode number of correct responses was 18….”

[Discussion of statistical test.]

“It was expected that the participants would be able to correctly classify more than 12 of the 24 strings in phase two, so the result is not especially surprising. Still, the data provide a striking example of implicit learning. ”


Infusing qr across the curriculum

Infusing QR Across the Curriculum

“…numeracy is not something mastered in a single course….Thus quantitative material needs to permeate the curriculum, … so that students have opportunities to practice their skills and see how useful they can be in understanding a wide range of problems.”

-Derek Bok (2006)

“…authentic and enduring learning…can rarely succeed one course at a time.”

-Lee Shulman (1997)


Infusing qr across the curriculum1

Infusing QR Across the Curriculum

“If [QR] remains the responsibility solely of mathematics departments—especially if it is caged into a single course such as ‘Math for Liberal Arts’—students will continue to see [QR] as something that happens only in the mathematics classroom.”

-Lynn Steen

Achieving Quantitative Literacy


Infusing qr across the curriculum2

Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant


Infusing qr across the curriculum3

Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant


Infusing qr across the curriculum4

Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant


Infusing qr across the curriculum5

Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant


Infusing qr across the curriculum6

Infusing QR Across the Curriculum

Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant


Infusing qr across the curriculum7

Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant


Infusing qr across the curriculum8

Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant


Infusing qr across the curriculum9

Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant


Infusing qr across the curriculum10

Infusing QR Across the Curriculum

Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio)

Legend

Green = Math & Natural Sciences

Blue = Social Sciences

Red = Other

Dark = Centrally Relevant

Medium = Peripherally Relevant

Light = Irrelevant


Infusing qr across the curriculum11

Infusing QR Across the Curriculum

  • 30% of gen ed papers are centrally QR relevant

    • Of these, nearly 50% are outside natural sciences

  • Another 25% are peripherally so

    • Of these, 90+% are outside natural sciences and more than half are in Arts, Lit, & Humanities


Infusing qr across the curriculum12

Infusing QR Across the Curriculum

By QR relevance, what fraction of papers use QR little/none, some, and extensively?


Infusing qr across the curriculum13

Infusing QR Across the Curriculum

“Weasely” examples:A) “The downtown areas of large cities are often home to copious franchises.”

B) Recall Daycare Paper


Our curricular response

Our Curricular Response


Our curricular response1

Our Curricular Response


Our curricular response2

Our Curricular Response


Our curricular response3

Our Curricular Response


Our curricular response4

Our Curricular Response

Disciplines Currently Represented:

American Studies

Biology (2)

Chemistry

Economics (5)

Environmental Science

English

Fine Arts

Geography (2)

Geoscience

Mathematics (12)

History (2)

Physics

Political Science (8)

Psychology (4)

Sociology (4)


Our curricular response5

Our Curricular Response

Coming Soon!:

Anthropology

Biology

Classics

Economics (2)

English

French (2)

History (3)

Political Science (4)

Psychology

Religion

Sociology (2)


Our curricular response6

Our Curricular Response

Example 1: An Ill-Structured Physics Lab

Example 2: Intro to Lintina/o Studies Paper

Example 3: History Paper on India’s Colonial Census

Example 4: Exploring and Architectural Remodel


Our curricular response7

Our Curricular Response

Example 5: Environmental History Paper

Example 6: Development Economics Paper

Example 7: Native-American Literature Analysis


Our curricular response8

Our Curricular Response

Suggestions for Assignment Creation:

Role

Audience

Format

Task

Task as

Ill-Structured

Problem


  • Login