1 / 43

Educational Standard Correlation Services for Digital Libraries

Educational Standard Correlation Services for Digital Libraries. Holly Devaul Jonathan Ostwald John Weatherley Digital Learning Sciences UCAR. Anne Diekema Jennifer Bailey CNLP Syracuse University. Presentation Overview.

odele
Download Presentation

Educational Standard Correlation Services for Digital Libraries

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Educational Standard Correlation Services for Digital Libraries Holly Devaul Jonathan Ostwald John Weatherley Digital Learning Sciences UCAR Anne Diekema Jennifer Bailey CNLP Syracuse University

  2. Presentation Overview • We will discuss a suite of educational standards assignment and alignment services that support cataloging and discovery of educational standards information in the NSDL • First half - how these services work, scenarios for use, and expected performance levels based on recent evaluations • Second half - more technical in nature; aimed at software engineers and web developers focusing on how the APIs work

  3. CASAA- Computer Assisted Standards Assignment and Alignment • NSF NSDL Service project with two goals • Support the assignment of standards to learning resources • Content Assignment Tool (CAT) • Provide a cross-walk between the different state and national standards • Standards Alignment Tool (SAT)

  4. Motivation • No Child Left Behind Act (NCLB) • Teachers must document how their teaching is aligned to standards • States required to develop assessments that evaluate student achievement relative to standards • Teachers want access to standards information for accountability reporting • Opportunity for digital libraries to provide this service

  5. Content Assignment Tool (CAT) • Natural Language Processing technology assesses resource content, suggests relevant educational content standards • Semi-automatic approach • CAT suggests relevant standards to the user • User makes final determination • System learns from vetted assignments • Web tool plus REST API for easy integration with other applications

  6. Web-based access to CAThttp://cat.nsdl.org:8280/casaa User name- guest Password- p

  7. Integration with theDigital Collection System (DCS) • Accesses resource content via URL • Utilizes data from catalog record • -url, grade level • -description, keywords (optional) • Is configured to query only NSES

  8. Suggestions provided in context of cataloging

  9. Visual cues help track suggested and human selected standards – human makes final decision on what to assign Standards Selected but not suggested Suggestion Selected Suggestion

  10. CAT Evaluation • Compared human-assigned NSES standards to CAT suggested standards • Research questions: • Does the CAT Service return standards that a human would choose for assignment? • Do the standards human catalogers have chosen occur near the top of the ranked list of CAT Service suggestions? • Does the inclusion of metadata improve standards assignment via the CAT Service?

  11. National Science Education Standards (NSES) • Hierarchical in structure, classified by grade band and content area and can be grouped as follows: • inquiry - (nature of science; Standard A) • subject - (life, physical, Earth and space science; Standards B, C and D) • applied - (science and technology, personal and social perspectives, and the history and nature of science; Standards E, F, and G).

  12. Does the CAT Service return standards that a human would choose for assignment? • CAT Service suggestions include 28-51% of the standards humans assigned • This offers a time savings in the initial assignment of the standards by presenting the cataloger a smaller subset of standards to choose from, that includes a substantial number of the preferred standards.

  13. Do the standards human catalogers have chosen occur near the top of the ranked list of CAT Service suggestions? • Analysis was conducted to assess the placement of the human-assigned standards in the ranked list of suggestions from CAT • If metadata included, 83% of the standards suggested at rank 1 were also chosen by a human; then drops off quickly • Subject standards performed better than applied or inquiry • We speculate that the language used in the resource corresponds better with subject standards than with applied or inquiry which require more subjective interpretation

  14. Does the inclusion of metadata improve standards assignment via the CAT Service? • YES- if no metadata included, 60% of the standards at rank 1 were also chosen by a human • In addition, including metadata expands the utility of this tool to suggest standards for non-text-based resources that have descriptive metadata associated with them • See technical paper for full details of this analysis

  15. Standards Alignment Tool (SAT) • Dynamic automatic correlation of standards • Combination of various algorithms to find equivalent (correlated) standards • Allows great flexibility for indexing and search • Takes standard text or ASN standard PURL • REST API for easy integration with other applications/systems

  16. Web-based access to SAThttp://sat.nsdl.org:8380/casaa-sat/ User name- guest Password- p

  17. SAT-DLESE Discovery integration • Testbed collection with NSES standards cataloged at 4th level • Select state of interest to view corresponding state standards side-by-side • Proof-of-concept for technical integration; alignments not yet vetted

  18. http://sat.dlsciences.org

  19. State standards are displayed side-by-side national standards

  20. SAT Evaluation • Looked at NSES to state mapping for five states (OH, MA, MN, CO, NY) • NSES standards distributed proportionately across inquiry, subject and applied subgroups and across grade level (n=49) • Queried SAT for state alignment to NSES • Evaluated first 5 suggestions for each standard (strong, moderate, weak correlation) or none at all

  21. Preliminary analysis • Issues • Lack of equivalent state standards • Differing granularity of scope of content • A mechanism to better discriminate no match and multiple matches is needed • For this alignment (NSES-state) human mediation is still necessary (as with CAT) • Other alignment pathways (state-to-state) need to be evaluated to get the full picture • Additional machine learning may improve results

  22. Part Two – The Technical Bits • Aimed at software engineers and web developers who wish to utilize the API’s and integrate CAT and SAT functionalities with existing applications

  23. CAT API in DCS: Topics • CAT / DCS Interaction • The Educational Standards Hierarchy • API call: suggestedStandards • Constraining the search for suggestions • Other uses of API in DCS • RPC vs REST Services

  24. DCS Communicates with CAT via Web Service

  25. Leaf Selectable / Suggested Standards are leaf nodes Standards Hierarchy

  26. ASN Identifier • Benchmark suggestStandards request CAT search for suggestions guided by info supplied to the API Returns StandardsList suggested by the CAT Service

  27. Constraining Suggestions • author – (NSES) • topic – (Science) • maxResults – (10) • startGrade • endGrade • keywords • Description • (weighted) keywords

  28. RESTRepresentational State Transfer

  29. CAT Service Response

  30. CAT API Requests • suggestStandards – Suggests aligned standards for a specific lesson plan • saveStandards – Save standards to a specific assignment • getStandard – Retrieve the details for a specified standard • getStandards – Retrieve the details for a specified list of standards

  31. CAT API Requests (continued…) • removeStandards – Remove standards from a specific assignment • getAssignments – Retrieve the standards for a specific assignment • getAllAssignments – Retrieve all assignments (including the standards for each assignment) for a specified user • getAuthors – Retrieve all assignments (including the standards for each assignment) for a specified user • getTopics – Retrieve a list of all topics available in CAT

  32. SAT Service: Topics • DLESE use case: Display state standards that are aligned to NSES • API call: suggestedStandards • Displaying the suggested standards in the DLESE interface • SAT API overview • Development tips

  33. DLESE Request for State Standards • User selects a state in the DLESE UI • DLESE provides ASN identifiers for NSES standards • suggestStandards request accepts ASN identifiers or text • SAT returns a list of suggested standards as XML • Response contains text suitable for display in the UI • DLESE UI displays the standards to the user

  34. DLESE Communicates with SAT via Web Service

  35. NSES Standards Supplied by DLESE, State Standards Returned by SAT State standards returned by SAT service as text NSES standards supplied by DLESE as ASN Ids

  36. suggestStandards request • One or multiple standards can be supplied in the request • Multiple standards handled one of two ways: • Combine the text of all the standards into a single query and return a single result set - or - • A query for each standard and return as many result sets as there are standards • DLESE used the latter • DLESE has multiple standards cataloged for each resource • Display best match for each standard • Supply grade range to refine results

  37. SAT suggestStandards Response

  38. DLESE Display • State standards displayed side-by-side with NSES standards • When a state is selected by the user, request is made asynchronously using AJAX • Provides time for SAT to respond and gives the user feedback with a spinner graphic • Selected state is remembered throughout the user’s session

  39. SAT API Requests • suggestStandards – Suggests aligned standards for specific standard • saveStandards – Save standards to a specific assignment • removeStandards – Remove standards from a specific assignment • getAssignments – Retrieve the standards for a specific assignment • getAllAssignments - Retrieve all assignments (including the standards for each assignment) for a specified user

  40. SAT API Requests (continued…) • getAuthors - Retrieve a list of all authors (states) available in SAT • getTopics - Retrieve a list of all topics available in SAT • getStandard - Retrieve the details for a specified standard • getStandards - Retrieve the details for a specified list of standards

  41. Development Tips • Use AJAX to retrieve and display standards at point of request • Retrieve and cache standards alignments ahead of time if needed • Use JSP/PHP to retrieve and display the results in a Web setting

  42. REST Service URLs • CAT http://cat.nsdl.org:8280/casaa/service.do • SAThttp://sat.nsdl.org:8380/casaa-sat/service.do

  43. Contact us for more information • Anne Diekema diekemar@syr.edu • Holly Devaul devaul@ucar.edu • Jonathan Ostwald ostwald@ucar.edu • John Weatherley jweather@ucar.edu

More Related