The item writing process for access for ells online involvement by esl teachers
Download
1 / 40

The Item-Writing Process for - PowerPoint PPT Presentation


  • 343 Views
  • Updated On :

The Item-Writing Process for ACCESS for ELLs™ Online Involvement by ESL Teachers. Jim Bauman Center for Applied Linguistics Washington, DC. January 2005 Illinois 28th Annual Statewide Conference for Teachers Serving Linguistically and Culturally Diverse Students. Presentation Outline.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The Item-Writing Process for' - Anita


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The item writing process for access for ells online involvement by esl teachers l.jpg

The Item-Writing Process forACCESS for ELLs™ Online Involvement by ESL Teachers

  • Jim Bauman

  • Center for Applied Linguistics

  • Washington, DC

January 2005

Illinois 28th Annual Statewide Conference for Teachers Serving Linguistically and Culturally Diverse Students

1


Presentation outline l.jpg
Presentation Outline

  • The WIDA Assessment Framework

  • Structure of ACCESS for ELLs™

  • The Item Writing Course

2


Wida s foundation l.jpg
WIDA’s Foundation

Professional

Development

Improved

teaching

Standards

AssessmentAccountability

Higher levels

of learning

Clear expectations for students and schoolsMotivation towork hard

Adapted from:

National Research Council, 1999

Testing, Teaching, and Learning:

A Guide for States and School Districts

3


Wida enhanced assessment system l.jpg
WIDA Enhanced Assessment System

ELP StandardsClassroomAssessment

ELP StandardsLarge-scaleAssessment

State Content

Standards

Lesson Plan

Specifications

Specifications

AssessmentFramework

Test Blueprint

StandardsTask Blueprint

Classroom Assessments

English LanguageProficiency Test

(ACCESS for ELLS)

Alternate Assessment

of Academic Achievement

(Alternate ACCESS)

4


Organization of the wida standards l.jpg
Organization of the WIDA Standards

Frameworks for Classroom & Large-Scale Assessment (2)

English Language Proficiency Standards (5)

SI

LA

MA

SC

SS

Language Domains (4)

Grade Level Clusters (4)

L

R

W

S

Language Proficiency Levels (5)

K-2

3-5

6-8

9-12

Model PIs are the lowest level ofexpression of the standards

Model Performance Indicators (>800)

1

2

3

4

5

5


Wida standards examples l.jpg
WIDA Standards Examples

Social Studies, Listening, 6-8

English language learners process, understand, interpret, and evaluate spoken language in a variety of situations.

Level 1: Identify icons on maps or graphs from oral statements.

Level 3: Categorize resources or products of regions from oral descriptions.

Level 5: Draw conclusions about resources or products in various regions based on oral descriptions.

Social & Instructional, Writing, 6-8

English language learners engage in written communication in a variety of forms for a variety of purposes and audiences.

Level 4: Suggest ideas for making changes in school, such as rearranging a schedule or adding subjects.

Language Arts, Reading, 6-8

English language learners process, interpret, and evaluate written language, symbols, and text with understanding and fluency.

Level 5: Draw conclusions from explicit and implicit text.

6


Aligning state content standards with wida elp standards l.jpg
Aligning State Content Standards with WIDA ELP Standards

MIDDLE/JUNIOR HIGH SCHOOL

Illinois State Social Science Standards

STATE GOAL 17: Understand world geography and the effects of geography on society, with an emphasis on the United States.

17.A.3b Explain how to make and use geographic representations to provide and enhance spatial information.

17.B.3a Explain how physical processes including climate, plate tectonics, erosion, soil formation, water cycle, and circulation patterns in the ocean shape patterns in the environment and influence availability and quality of natural resources.

17.C.3b Explain how patterns of resources are used throughout the world.

WIDA Social Studies Standards — Listening

English language learners process, understand, interpret, and evaluate spoken language in a variety of situations.

Level 1: Identify icons on maps or graphs from oral statements.

Level 3: Categorize resources or products of regions from oral descriptions.

Level 5: Draw conclusions about resources or products in various regions based on oral descriptions.

7


Presentation outline8 l.jpg
Presentation Outline

  • The WIDA Assessment Framework and Standards

  • Structure of ACCESS for ELLs™

  • The Item Writing Course

8


Structure of access for ells l.jpg

Grade Level and Tier

Structure of ACCESS for ELLs

K

1-2

3-5

6-8

9-12

A (adaptive)

A B C

A B C

A B C

A B C

  • Domains

Listening — group admin, machine scored

Reading — group admin, machine scored

Speaking — individual admin, adaptive, TA scored

Writing — group admin, rater scored

  • Forms

100 (roll-out Spring 2005)

999 (used to produce screener)

200 (roll-out Spring 2006)

9


Theme and item structure l.jpg
Theme and Item Structure

  • Theme

Item Model

Item Stimulus

Question and Response Options

10


Expression of the thematic folder l.jpg
Expression of the Thematic Folder

  • Orientation

    • Contextualizes the items

    • Engages background knowledge

  • Stimulus — Text and/or Graphic

    • Presents new information

    • Organizes information visually

    • Reduces memory load

  • 3 or more items at same or scaffolded proficiency levels

    • Item levels determined by specs

    • Address single strand of PIs

11


Sample listening theme l.jpg
Sample Listening Theme

Script: Look at the map. It shows some of the important economic activities in different regions of modern Russia. Each symbol in the legend shows a different economic activity. A fish indicates fishing, a factory heavy industry, logs forestry, wheat agriculture, a pick and shovel mining, and an oil rig petroleum.

The Russian Economy

12


Sample listening items l.jpg
Sample Listening Items

1

Script: Find the symbol for oil.

Script: Russia has many kinds of natural resources. These include coal and iron ores from mining, oil from drilling, and wood from logging. Which region in Russia has the greatest variety of natural resources?

3

Urals

Western Siberia

Eastern Siberia

Far East

Script: The part of Russia from Siberia east is extremely cold for much of the year and is sparsely populated. The western part of the country, particularly in the south, has a milder climate and a longer growing season. Almost 80 percent of the population lives in the western part.

Since factories require a large labor force to run them but they also need raw materials and energy, where would you probably find the greatest concentration of Russia’s heavy industry?

5

In the southern part of the country where food is plentiful

In the northern part of the country where the population is large

In the eastern part of the country where raw materials are abundant

In the middle part of the country where raw materials and people are abundant

13


Sample writing theme l.jpg
Sample Writing Theme

The principal at your school has decided to change the prices of some foods in the school cafeteria. Some students like the changes and some do not.

14


Sample writing item l.jpg
Sample Writing Item

Look at the new cafeteria price list. Suggest changes you would make to the prices for these or other foods.

Write a short paragraph of at least 8 sentences. Write about changes you would make to cafeteria prices.

15


Goals of an item specification l.jpg
Goals of an Item Specification

Social Studies, Listening, 6-8, Level 1

  • State the assessment objective

    • By grade

    • By PI

  • Describe the item stimulus

    • Use of graphics/text

    • Mode of presentation to test taker

  • Describe the test taker’s task

    • Method of selecting a correct response

    • Method of constructing a scorable response

    • Properties of response options

16


Life cycle of a test item l.jpg
Life Cycle of a Test Item

  • Receive theme & items from item writer

  • Submit items to external review

  • Commission illustrations

  • Check thematic folder for internal consistency

  • Submit folder and items to bias review

  • Layout folder in test form

  • Field test to establish item properties

  • Operationalize for one year

  • Retire the folder and items

17


What the item writer does l.jpg
What the Item Writer Does

R e v i e w &

RevIse

Read the scopeof the itemin the specsstatement

Read the stimulusdescription & identify an appropriate content area

Write ordescribe the themestimulus and/ormodel

Write the itemstimuli orprompts

Read the taskdescription &write responseoptionsfor the items

18


Item creation review l.jpg
Item Creation & Review

Specifications

Raw Items

Item Writers

InternalReviewers

Cleaned Items

Bias & ContentReviewers

ApprovedItems(ready for field testing)

19


Raw item l.jpg
Raw Item

  • Selected-Response Item

    • Language Arts, Reading, Grade 6-8, Level 5

Orientation

In your test booklet are some sentences that tell why something happened in the story I will read.

Theme Stimulus

Leo and his father went to the grocery store. They bought food for a party they are having. At the grocery store, they picked out a cake and some ice cream. When they got home, Leo’s father put the cake on the table in the kitchen and walked away. Then, Leo’s dog, who had not eaten all day, came into the kitchen and ate the whole cake!

Specification(in part)

Stimulus

None

Why did Leo’s dog eat the cake?

Question

Options

A. Because he was hungry

B. Because he was a bad dog

C. Because Leo told him to eat it

D. Because he was thirsty

A. Because he was hungry

Key

20


Cleaned item l.jpg
Cleaned Item

Orientation

The Brown family is going to have a party for their friends and neighbors to celebrate the Fourth of July.

Item is reconceived as part of a thematic folder with items at levels 3, 4, and 5 (only level 5 is shown here)

Theme Stimulus

Generic picture of a family making preparations for a party.

Leo and his father went to the grocery store to buy food for the party, but they forgot to feed their dog Rambo before they left. At the grocery store, they picked out a cake, some ice cream, and soda. When they got home, Leo put the food on table in the kitchen and went to help his father. When he got back, he was horrified to see that Rambo had his nose in the cake, happily eating away.

Item Stimulus(Level 5)

Why did Rambo eat the cake?

Question

Options

A. Because he was hungry

B. Because he was happy

C. Because Leo took him to the grocery store

D. Because Leo did not buy him dog food

A. Because he was hungry

Key

21


Approved item l.jpg
Approved Item

Orientation

The Rodriquez family is going to have a party for their friends and neighbors to celebrate Cinco de Mayo, an important holiday in Mexico.

Item after review by content and bias committee.

Theme Stimulus

Generic picture of a family making preparations for a party. Family members should look Hispanic.

Leo and his father went to the grocery store to buy food for the party, but they forgot to feed their dog Cabo before they left. At the grocery store, they picked out a cake, some ice cream, and soda. When they got home, Leo put the food on table in the kitchen and went to help his father. When he got back, he was horrified to see that Cabo had his nose in the cake, happily eating away.

Item Stimulus(Level 5)

Why did Cabo eat the cake?

Question

Options

A. Because he was hungry

B. Because he was happy

C. Because Leo took him to the grocery store

D. Because Leo did not buy him dog food

A. Because he was hungry

Key

22


The review process l.jpg
The Review Process

  • Check for fit to PIs

    • Does length of stimulus match spec?

    • Does linguistic complexity of stimulus match spec?

    • Is vocabulary appropriate to spec?

  • Check response options properties

    • Is there only one correct response option?

    • Are distractors motivated and equally attractive?

    • Are response options balanced in length & complexity?

  • Check for inter-item balance & integrity

    • Are items consistent with theme?

    • Are items independent?

  • Check for content accuracy

  • Check for language, culture, and gender bias

23


Presentation outline24 l.jpg
Presentation Outline

  • The WIDA Assessment Framework and Standards

  • Structure of ACCESS for ELLs™

The Item Writing Course

24


Course goals l.jpg
Course Goals

  • Understand and apply established principles of language test construction, including notions of test validity and reliability

  • Develop test items which accord with guidelines established for determining item difficulty, bias and authenticity

  • Develop test items which accurately address a set of predetermined item specifications (the WIDA specifications), including writing test prompts and scoring rubrics

  • Develop test items across the range of item formats designated by the WIDA specifications

  • Develop test items that target specific levels of English language proficiency

  • Develop test items that are aligned with the functional language requirements of social and classroom language and of the content areas: English language arts, mathematics, science and social studies.

25


Course work requirements l.jpg
Course Work Requirements

  • Study and apply the specifications for the participant’s designated grade level

  • Write test items in each of the four language domains for the designated grade level

  • Communicate with the class on substantive issues via discussion boards

  • Communicate with the class on procedural issues via chat room

  • Interact with other members of designated grade level

  • Take and pass quizzes and final exam

26


Course participants l.jpg
Course Participants

  • Lead Instructor: Jim Bauman

  • Associate Instructors: Jessica Motz & Ellen Parkhurst

  • Teaching Assistants: George Motz (1-2), Lee Gough (3-5), Jennifer Himmel (6-8), Ellen Daniels (9-12)

  • Active Participants (28)

    From IL:

    Marion Flaman—SD54Michael Safina—CCSD59Diane Wilkin—SD99Delia Rodriguez—CUSD300Irene Pelc—CCSD15Deb Grant—HSC220

  • Auditors (5)

27


Course grading structure l.jpg
Course Grading Structure

  • Item submission 60%

  • Participation in chat sessions and discussions 20%

  • Quizzes 20%

Successful completion of course requirements at an 80% level earns 2 university credits from the University of Wisconsin

28


Course home page l.jpg
Course Home Page

CourseMenu

29



Structure of a module l.jpg
Structure of a Module

  • Each module contains:

  • Readings

  • Self-Paced Exercises

  • Interactive Exercises

    • Discussions

    • Chats

  • Assignments

  • Quizzes

31





Completing an assignment l.jpg
Completing an Assignment

Specification

  • Download appropriate set of item specs from Documents area

  • Download data entry form from Documents

  • Conceive & research folder and items

  • Complete item data entry form

  • Submit completed form to Dropbox

  • Revise per instructor’s feedback

  • Resubmit to Dropbox

BlankData Entry Form

Content Standards

CompletedData Entry Form

Dropbox

35




What teachers got as item writers l.jpg
What Teachers Got as Item Writers

  • University credit or CEUs

  • Direct experience with on-line learning

  • Professional development on the WIDA standards and writing items for large-scale assessments

  • Informed discussion on how to apply WIDA standards to the curriculum

  • Professional contacts with other teachers across the consortium

  • The deep gratitude of the ACCESS for ELLs™ development staff!!

38


What wida got from teacher item writers l.jpg
What WIDA Got from Teacher Item Writers

  • Approximately 1600 raw items

  • A nucleus in each state of WIDA informed staff

  • A pool of talent to serve as test coordinators, administrators, and future item reviewers

  • Valuable insight on teachers’ perspectives on standardized testing

  • Important clues on how to build professional development to apply the WIDA framework in the classroom

39



ad