Eric J. Pyle James Madison University

1. A Faculty-Student Dialogue on Assessment: Rubrics as Vehicles for Communicating Learning Expectations in the Field • Eric J. Pyle • James Madison University

2. Assessment vs. Evaluation • Assessment involves comparing information gathered from subjects relative to some established goal or objective (Kizlik, 2009). These goals, objectives, or outcomes are set in advance, and should be clear to both instructors and students. • Evaluation allows the establishment and communication of the worth of an activity (Kizlik, 2009). This worth can be determined by the extent to which decisions of instructional approaches, arrangements, organization, etc., are effective in aiding students into reaching the desired outcomes.

3. Goals vs. Objectives • Goals reflect the overall outcome expected of an learning activity, such a capacity to generate geologic descriptions of areas; • Objectives are the specific means by which students can demonstrate that they have met the goal expected of them, such as generating a geologic map, cross-section, and lithologic descriptions for a given area.

4. Objectives • Objectives consist of three parts: • Condition refers to the context of the learning experience, the setting, the materials, available or to be used; • Performance is a statement of what students are supposed to do, a description of the task or demonstration; • Criterion is the indication that the particular objective has been met, and to what degree it has been met. • Example: Given a base map and aerial photograph of an area, the student will produce a geologic map and cross-section that describes the relative sizes and orientations of the lithologies and structures for the region. Chiappetta & Koballa, 2006

5. Assessments and Objectives • Knowledge – specific knowledge, recall, terminology, etc.; • Skills – specific performances, equipment, tool-use, etc. • Dispositions – scientific habits of mind, persistence, question-posing, etc. • The struggle is to ensure that the entire range of learning described in the goals is expressed through appropriate objectives and related assessment tasks – validity.

6. Types of Assessment • Objective/Knowledge items - multiple choice, fill-in-the-blank; uses binary, nominal criterion • Prescribed/Lexical tasks - problems, short answers; procedures as well as the task are of value; uses ordinal level criterion measure; • Open-Ended/Prototype tasks - essays, artifacts; uses interval level criterion measure on 0-100 scale. • The struggle in each form is to provide a clear and consistent determination of mastery of the related objectives – reliability

7. The special case of field learning • Expectations are not just limited expressions of knowledge but the application of that knowledge; • Expectations include acquisition, demonstration, and mastery of skill sets; • Expectations include consistency, persistence, attention to detail, and synthesis of working hypotheses.

8. The special case of field learning:Knowledge Based on Bloom (1956)

9. The special case of field learning:Skills Based on Dave (1975)

10. The special case of field learning:Dispositions Based on Krathwohl, Bloom, & Masia (1973)

11. Combining Domains in Assessments Based on Smith (1995), Lawrence & Margolis (1999), Murphy (2002), and Sibley (2005)

12. Application: Field Notes Rubric • Condition: note-taking of field situations, with general expectations of description and supporting other documentation, such as maps or lithologic descriptions

13. Where is the Dialogue? • Exchange begins with sharing the rubrics ahead of time • Criterion can be stated a priori by instructors, which allows reliable measurement; or • Negotiated with students, which allows for closer validity of measurement to particular or unique contexts. • Student first (and continuing) response is usually a request for clarification • Reply is finer grained – not “what do you want,” but “what do you mean by,” allowing more specific examples to be generated. • Next exchange is the assignment of scores • Scores are broken down by the level of mastery for each dimension or objective; • Narrative feedback becomes focused on stated criteria. • Next exchange is student formative response – what do they do next time to improve, or in some cases, revision of prior submission; • Less prominent is the student of response of “why did I get this grade,” when subscores relative to descriptors are provided.

14. Other Applications in Field Courses:Maps and Cross-Sections

15. Other Applications in Field Courses:Lithologic Descriptions, Memoirs

16. Uses and Extensions • These rubrics have been used with two successive JMU field courses; • In general, their use allows for quicker and more consistent turn-around of student assignments; • But continued discussion among participating faculty is needed to fine-tune language relative to expectations; • The Knowledge-Skills-Dispositions format is also employed in student evaluations of instruction, specific to the individual assignments in the field course. Data analysis is on-going; • Rubrics are frequently uses on assignments in BA-Earth Science classes at JMU; • A general rubric is in use with the JMU program evaluation – Students participate in a performance task, with differential expectations based on student standing in program.

17. References • Bloom, B. S. (1956). Taxonomy of educational objectives, handbook I: The cognitive domain. New York: David McKay Co., Inc. • Chiappetta, E. L. & Koballa, T. R. (2006). Science instruction in the middle and secondary schools: Developing fundamental knowledge and skills for teaching, 6th ed. New York: Allyn & Bacon. • Dave, R. H. (1975). Developing and writing behavioral objectives. R. J. Armstrong, ed. Educational Innovators Press. • Kizlik, B. (2009) Measurement, Assessment, and Evaluation in Education. Website: http://www.adprima.com/measurement.htm, retrieved 3/10/09. • Krathwohl, D. R., Bloom, B. S., and Masia, B. B. (1973). Taxonomy of education objectives, the classification of educational goals, handbook II: Affective domain. New York: David McKay Co. Inc. • Lawrence, S., and Margolis, E. (1999). Concepts and cognitive science. In E. Margolis and S. Lawrence (eds.), 3-83. Concepts. Cambridge, MA: MIT Press. • Murphy, G. L. (2002). The big book of concepts. Cambridge, MA: MIT Press. • Pyle, E. J. (2009). The evaluation of field course experiences: A framework for development, improvement, and reporting. In S. J. Whitmeyer, D. Mogk, & E. J. Pyle (Eds.) Field Geology Education: Historical Perspectives and Modern Approaches. GSA Special Paper 461. Boulder, CO: Geological Society of America. • Pyle, E. J., & Brunkhorst, B. (2009). Developing and applying the knowledge, skills, and dispositions needed for effective Earth science teaching. In A. Collins & N. Gillespie (Eds.) The Continuum of Secondary Science Teacher Preparation: Knowledge, Questions, and Research Recommendations. Rotterdam: Sense Publishers. • Sibley, D. F. (2005). Visual abilities and misconceptions about plate tectonics. Journal of Geoscience Education, 53(4), 471-477. • Smith, E. (1995). Concepts and categorization. In E. Smith and D. Osherson (eds.), Thinking: An invitation to cognitive science, vol. 3., 3-33. Cambridge, MA: MIT Press.