1 / 30

Workshop on Quality/Selectivity of the DLESE Collections

This workshop discusses the criteria and procedures for approving resources for inclusion in the DLESE Broad Collection. It explores issues of relevance, quality, and potential problems that may arise.

spartain
Download Presentation

Workshop on Quality/Selectivity of the DLESE Collections

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Workshop onQuality/Selectivity of the DLESE Collections • Framing the Question • History of the Discussion Kim Kastens, June 30, 2003

  2. Framing the Question • DLESE has Broad Collection and a Reviewed Collection. We are (mostly) talking about the Broad Collection.

  3. Framing the Question • Resources enter the DLESE Broad Collection via two routes: • Individually, via the DLESE Cataloging Tool (the “Community Collection”) • As part of an aggregated or themed collection, a collection accessioned into DLESE in its entirety. • We are concerned with quality and relevance of resources entering via both routes.

  4. Framing the Question • Anyone can submit a resource to DLESE via the cataloging tool, which is an open set of web forms. • This has given rise to concerns that “junk” could get into the DLESE Collections.

  5. Framing the Question • This workshop needs to make recommendations on two issues: • What should be the criteria by which resources are approved for inclusion in the DLESE Broad Collection? • What should be the procedures by which these criteria are implemented?

  6. Framing the Question: Criteria • Resources submitted for the DLESE Broad collection currently must meet two criteria: • The resource is relevant to Earth System Education • The resource works (i.e. it has no conspicuous bugs).

  7. Framing the Question: Criteria • Other possible criteria that have been suggested for the DLESE Broad Collection: • No cost or low cost for educational users • Resource is in English • No commercial message • No intrusive advertising • No blatant religious message • No blatant political message • No blatant errors of fact • Educational effectiveness • Well documented

  8. Framing the Question: Procedures • With respect to procedures, we have two issues: • By what process shall we identify problematic resources? • What shall we do when we find a problematic resource?

  9. Framing the Question: Procedures • By what process shall we identify problematic resources? • Ask the resource contributor (current system)? • Screening by the community? • Screening by paid staff?

  10. Framing the Question: Procedures • What shall we do when we find a problematic resource? • Exclude it from the collection? • Include it in the Broad Collection but annotate it?

  11. This is what an annotation might look like in the Discovery System:

  12. Framing the Question: Procedures

  13. History of the Discussion • Coolfont: August 1999 • Collections Policy drafted • There shall be a Reviewed Collection and an “Unreviewed” Collection • Collection Committee established

  14. History of the Discussion • Coolfont: August 1999 (cont’d) • Rationale for Reviewed Collection: • Users’ Perspective: “…. recognized, efficient source for quality teaching and learning materials.” • Creators’ Perspective:“…. a recognized stamp of professional approval at the level of publication in a peer-reviewed journal.” • Rationale for the “Unreviewed” Collection: • “Users are seeking materials on a huge range of topics. The DL provides added value by being inclusive while providing powerful search and classification capability.”

  15. History of the Discussion • Coolfont: August 1999 (cont’d) • Criteria for Reviewed Collection: • Accuracy, as evaluated by scientists • Importance/significance • Pedagogical effectiveness. • Well documented. • Ease of use for students and faculty • Inspirational or motivational for students • Robustness/sustainability

  16. History of the Discussion • Coolfont: August 1999 (cont’d) • No Criteria established for “Unreviewed” Collection • After debate, it was decided that there would be a human-mediated step between submission of resource and ingestion into library.

  17. History of the Discussion • Spring 2000: Academic Career Recognition Task Force Web Survey • Seven selection criteria for the Reviewed Collection met approval of prospective DLESE users, resource creators, and department Chairs.

  18. History of the Discussion • Mid-late 2000: Collecting began • DPC: testbed collection for exercising metatdata framework • Montana State: Dave Mogk & students • Foothill College: Chris DiLeonardo & students

  19. History of the Discussion • October 2000: Collections Meeting at Boulder: • DLESE Community Cataloger tool introduced to non-DPC collecting groups (AGI, Montana State, others?)

  20. History of the Discussion • November 2000: Steering Committee Meeting at Lamont: • Contentious discussion about “filters” at the gateway to the Broad Collection • Agreement on only two of the discussed “filters”: (1) relevant to Earth System Education (2) “It works”, e.g. no conspicuous bugs • Contentious discussion of how to apply “filters”; clarity seemed to emerge when John Snow described a “holding tank” system used in his history group.

  21. History of the Discussion • Nov-Dec 2000: Steering Committee Meeting at Lamont (cont’d): • Meeting Minutes: • “The general concept of a 30-day public comment period on new resources was agreed to. This will allow a time for the community to review resources….” • “In the short term, partners collecting resources …. will review them to make sure they are appropriate • “….the Collections Committee, collection proposal team and the DPC will work together to investigate mechanisms for encouraging review….”

  22. History of the Discussion • February 2001 Collections Meeting: • Joint meeting of Collections Committee, “Collections Partners”, and Community Review System Editorial Review Board • DLESE Community Cataloging Tool open to the world • Collections Committee drafted Deaccession Policy

  23. History of the Discussion • February 2001 Collections Meeting (cont’d): • Collections Committee discussed “filters” at gateway to Broad Collection. Imperfect consensus: • Relevance Filter • Is the resource relevant to Earth System Science education? • Integrity Filter • Are there no blatant errors of fact in the resource? • Are there no blatant political, religious, or commercial messages in the resource? • Does it function reasonably; i.e., seem to be basically bug-free?

  24. History of the Discussion • April 2001 Steering Committee meeting at Biosphere 2: • Collections Committee/DPC Collections group presented fleshed out version of the “holding tank” or “provisional status” plan. • Many questions and issues. Who are reviewers? How mobilized and overseen? No $ to oversee the “army of filterers.” • No resolution.

  25. History of the Discussion • July-Aug 2001 Steering Committee meeting at Flagstaff: • 850 resources in library. Metadata QA streamlined. • “Mike Mayhew indicated a concern …about the broad collection. ….Where is the quality control in developing the collection? Do we dilute the value of library with variable quality?” • Holding tank idea revisited, in simpler form without “designated reviewers” • Action item: “Boyd …. will develop a draft proposal/set of guidines to implement a holding tank in which resources are discoverable in the system and identified as accessioned within a 30-day period with some mechanism to accept comments. The proposal for implementation will not include a designated reviewer”

  26. History of the Discussion • February 2002 Steering Committee meeting at Boulder: • Draft Collections Accession Policy presented • Revised throughout spring • DLESE oversight would be review of review process, rather than review of individual resources

  27. History of the Discussion • July 2002 Steering Committee & Annual meeting at Cornell: • Deaccession Policy approved • Interim Collection Accession Policy approved • First annotation service demo’d within DLESE • Faulker reported that NSDL content philosophy was: “Educational value …to be manifest in capabilities for annotation and selective filtering, rather than an accession threshhold” • Possibility raised that annotation option might be solution to ongoing dilemma about quality of DLESE Broad Collection.

  28. History of the Discussion • Fall 2002: • Sumner et al focus group study of Educators’ perceptions of Quality. • Best Practices for Resources summited to the DLESE Reviewed Collection begins to take shape.

  29. History of the Discussion • Spring 2003: • Ad hoc Collections group met in Boulder, worked on how to implement Interim Collections Accession Policy and on Pathways to Reviewed Collection document • June 13: 12 collections met documentation requirements to be accessioned as collections.

More Related