1 / 17

A Review of the Use of Knowledge Mapping for Assessment Purposes

This review examines the use of knowledge mapping for assessment purposes, presenting findings from empirical studies on scoring methods, reliability, and validity. Knowledge maps depict concept relationships through nodes and links, often employed for instructional goals and occasionally for assessment. The review synthesizes 38 studies, highlighting scoring approaches, high reliability achieved through trained raters, and challenges in human rating. Automated scoring methods are discussed as feasible solutions, promising reliable and immediate feedback while addressing the labor-intensive nature of traditional assessments.

nuncio
Download Presentation

A Review of the Use of Knowledge Mapping for Assessment Purposes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Review of the Use of Knowledge Mapping for Assessment Purposes Gregory K. W. K. Chung Eva L. Baker California Educational Research AssociationAnnual Meeting Rancho Mirage, CA – December 4, 2008

  2. Overview of Talk • Research questions • Methodology • Reliability • Validity • Conclusion

  3. Knowledge Maps • Node-link representation (nodes = concepts, links = relationships) • Typically used for instructional purposes • Sometimes used for assessment • Can be scored automatically leads to surface warming sunlight

  4. The Depression

  5. Research Questions • What are the scoring methods for knowledge maps? • What is the reliability and validity evidence? • What are the feasibility issues?

  6. Methodology • Prior reviews • Ruiz-Primo, M. A., & Shavelson, R. J. (1996). Problems and issues in the use of concept maps in science assessment. Journal of Research in Science Teaching, 33, 569-600. • Chung, G. K. W. K., Baker, E. L., Brill, D. G., Sinha, R., Saadat, F., & Bewley, W. L. (2003). Automated assessment of domain knowledge with online knowledge mapping. Proceedings of the I/ITSEC, 25, 1168–1179.

  7. Methodology • Criteria for inclusion in review • Empirical studies reported after Ruiz-Primo and Shavelson (1996) • Study reported technical information (reliability or validity information) • CRESST technical reports and CRESST-supported dissertations

  8. Scoring Approaches

  9. Referent-Based / SemanticReferent-Free / Semantic Propositions in map banking crisis —contributed to—> Depression Hoover —part of—> Depression unemployment —contributed to—> Depression New Deal —response to—> Depression etc.

  10. Results

  11. Sample • 38 studies • 23 affiliated with CRESST (UCLA, Stanford) • 15 from other universities • Studies reported • Scoring method • Reliability or validity coefficients

  12. Reliability • Rating of maps (by human or computer) • High reliability—raters can be trained to evaluate knowledge maps • alpha: .6 to .9 • g-coefficient: .8 to .9 • Constraining task yields highest reliability • Use a fixed set of concepts and links • Expert-map referent yields highest reliabilities

  13. Validity • Correlation with other measures of similar content • “Less conceptual” r: .4 to .5 • “More conceptual” r: .4 to .7 • Sensitive to knowledge differences • Experts >> Novices • Pre-instruction < Post-instruction

  14. Feasibility • Human ratings generally tedious, labor-intensive, and time-consuming • Only way to do with unconstrained tasks • Automated scoring feasible • High reliability, immediate feedback • Constrained task

  15. Conclusion • In general, knowledge maps are feasible, reliable, and sensitive to knowledge differences and instructional effects • Task format influences reliability • Feasibility an issue—human rating of knowledge maps can be labor-intensive and tedious • Use of an expert-criterion map yields the highest reliability • Automated scoring tractable with predefined concepts and links

  16. greg@ucla.edu

More Related