slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Workshop on Metadata Standards and Best Practices November 19-20 th , 2007 Session 4 The Data Documentation Initiative T PowerPoint Presentation
Download Presentation
Workshop on Metadata Standards and Best Practices November 19-20 th , 2007 Session 4 The Data Documentation Initiative T

Loading in 2 Seconds...

play fullscreen
1 / 41

Workshop on Metadata Standards and Best Practices November 19-20 th , 2007 Session 4 The Data Documentation Initiative T - PowerPoint PPT Presentation


  • 242 Views
  • Uploaded on

Workshop on Metadata Standards and Best Practices November 19-20 th , 2007 Session 4 The Data Documentation Initiative Technical Overview. Pascal Heus Open Data Foundation pheus@opendatafoundation.org http://www.opendatafoundation.org. Outline. XML refresher DDI Background DDI 1/2.x

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Workshop on Metadata Standards and Best Practices November 19-20 th , 2007 Session 4 The Data Documentation Initiative T' - halle


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Workshop on Metadata Standards and Best PracticesNovember 19-20th, 2007Session 4The Data Documentation InitiativeTechnical Overview

Pascal Heus

Open Data Foundation

pheus@opendatafoundation.org

http://www.opendatafoundation.org

outline
Outline
  • XML refresher
  • DDI Background
  • DDI 1/2.x
    • Status / Tools
  • DDI 3.0
    • Use cases
    • Need for tools
  • Conclusions / Q&A

Thanks to the DDI Alliance and GESIS

for slides on DDI

Open Data Foundation – IZA 2007/11

xml to the rescue
XML to the rescue!
  • XML is driving today’s web service oriented architecture of the Internet and Intranets
  • Using XML, we can capture, structure, transform, discover, exchange, query, edit and secure metadata and data
  • XML is platform & language independent and can be used by everyone
  • XML is both machine and human readable
  • XML is non-proprietary, public domain and many open tools exist
  • Domain specific standards are available!

Open Data Foundation – IZA 2007/11

xml technical overview
XML Technical Overview

Open Data Foundation – IZA 2007/11

what is ddi
What is DDI?
  • The Data Documentation Initiative (DDI) is an XML format for capturing metadata about survey data and register data
  • Data files may remain in their native formats (ASCII files which may be delimited or fixed-width) or may be captured as XML
  • It used to be designed to describe codebooks, and was mainly useful for data archives and libraries
    • Versions 1.*/2.*
  • Now, it can be used for any type of data collection
    • Version 3.0
    • Focus on survey instrumentation and microdata, but also can describe aggregates

Open Data Foundation – IZA 2007/11

background
Background
  • Concept of DDI and definition of needs grew out of the data archival community
  • Established in 1995 as a grant funded project initiated and organized by ICPSR
  • Members:
    • Social Science Data Archives (US, Canada, Europe)
    • Statistical data producers (including US Bureau of the Census, the US Bureau of Labor Statistics, Statistics Canada and Health Canada)
  • February 2003 – Formation of DDI Alliance
    • Membership-based alliance
    • Formalized development procedures

Open Data Foundation – IZA 2007/11

growth of the ddi structure
Growth of the DDI Structure
  • 2000 – DDI 1.0
    • Simple survey
    • Archival data formats
    • Microdata only
  • 2003 – DDI 2.0
    • Aggregate data (based on matrix structure)
    • Added geographic material to aid geographic search systems and GIS users

Open Data Foundation – IZA 2007/11

characteristics of ddi 1 0 2 0
Characteristics of DDI 1.0/2.0
  • Focuses on the static object of a codebook
  • Designed for limited uses
    • End user data discovery via the variable or high level study identification (bibliographic)
    • Only heavily structured content relates to information used to drive statistical analysis
  • Coverage is focused on single study, single data file, simple survey and aggregate data files
  • Variable contains majority of information (question, categories, data typing, physical storage information, statistics)

Open Data Foundation – IZA 2007/11

1 2 x schema
1/2.x Schema
  • Organized in 5 sections
    • docDsrc: information about the XML document itself: metadata preparation, version,
    • stdyDscr: detailed information about the survey
      • Title, year, coverage, sampling, data collection/cleaning, quality, contact, access policy,
    • fileDscr: describes files in the dataset
    • dataDscr: describes the data structure
      • Variable: name, label, code, summary statistics, definitions, literal question, interviewer instructions, weights, grouping, etc.
      • Cubes: aggregated data
    • othMat: additional documentation
  • See examples

Open Data Foundation – IZA 2007/11

limitations
Limitations
  • Treated as an “add on” to the data collection process
  • Focus is on the data end product and end users (static)
  • Limited tools for creation or exploitation
  • The variable must exist before metadata can be created
  • Producers hesitant to take up DDI creation because it is a cost and does not support their development or collection process

Open Data Foundation – IZA 2007/11

requirements for 3 0
Requirements for 3.0
  • Improve and expand the machine-actionable aspects of the DDI to support programming and software systems
  • Support CAI instruments through expanded description of the questionnaire (content and question flow)
  • Support the description of data series (longitudinal surveys, panel studies, recurring waves, etc.)
  • Support comparison, in particular comparison by design but also comparison-after-the fact (harmonization)
  • Improve support for describing complex data files (record and file linkages)
  • Provide improved support for geographic content to facilitate linking to geographic files (shape files, boundary files, etc.)

Open Data Foundation – IZA 2007/11

approach
Approach
  • Shift from the codebook centric model of early versions of DDI to a lifecycle model
    • providing metadata support from data study conception through analysis and repurposing of data
  • Shift from an XML Data Type Definition (DTD) to an XML Schema model
    • to support the lifecycle model, reuse of content and increased controls to support programming needs
  • Redefine a “single DDI instance” to include a “simple instance”
    • similar to DDI 1.*/2.* which covered a single study and “complex instances” covering groups of related studies.
    • Allow a single study description to contain multiple data products (for example, a microdata file and aggregate products created from the same data collection).
  • Incorporate the requested functionality in the first published edition

Open Data Foundation – IZA 2007/11

designing to support registries
Designing to support registries
  • Resource package
    • structure to publish non-study-specific materials for reuse
  • Extracting specified types of information into schemes
    • Universe, Concept, Category, Code, Question, Instrument, Variable
  • Allowing for either internal or external references
  • Providing comparison mapping
    • Target can be external harmonized structure

Open Data Foundation – IZA 2007/11

relationship to other standards
Relationship to Other Standards
  • Dublin Core
    • Mapping of citation elements
    • Option for DC namespace basic entry
  • ISO 19115 – Geography
    • Search requirements
    • Support for GIS users
  • METS
    • Designed to support profile development
  • OAIS
    • Reference model for the archival lifecycle
  • SDMX
    • Completely mapping to and from DDI NCubes
    • Designed to be used with registries
  • ISO/IEC 11179
    • Variable linking representation to concept and universe
    • Optional data element construct in ConceptualComponent that allows for complete ISO/IEC 11179 structure as a maintained item

Open Data Foundation – IZA 2007/11

development of ddi 3 0
2004 – Acceptance of a new DDI paradigm

Lifecycle model

Shift from the codebook centric / variable centric model to capturing the lifecycle of data

Agreement on expanded areas of coverage

2005

Presentation of schema structure

Focus on points of metadata creation and reuse

2006

Presentation of first complete 3.0 model

Internal and public review

2007

Vote to move to Candidate Version

Establishment of a set of use cases to test application and implementation

2008

March: anticipated vote to publish DDI 3.0

Development of DDI 3.0

Open Data Foundation – IZA 2007/11

xml schemas and 3 0 modules one is not necessarily the other
XML Schemas and 3.0 Modules (one is not necessarily the other)
  • XML Schemas
    • Each .xsd file is a xml schema
    • Some xml schemas are modules
    • Some xml schemas are substitution sets or “sub-modules”
    • Some xml schemas simply contain elements that are used by multiple schemas or may require more frequent updates
    • Some xml schemas are “external”

Open Data Foundation – IZA 2007/11

xml schemas and 3 0 modules one is not necessarily the other1
XML Schemas and 3.0 Modules (one is not necessarily the other)
  • Modules
    • Reflect closely related sets of information similar to the sections of DDI 1.*/2.* DTD
    • Modules can be held as separate XML instances and be included in a large instance by either inclusion or reference
    • All modules are maintainable, identifiable packages
    • Each module has its own XML namespace

Open Data Foundation – IZA 2007/11

xml schemas
archive

comparative

conceptualcomponent

datacollection

dataset

dcelements

DDIprofile

ddi-xhtml11

ddi-xhtml11-model-1

ddi-xhtml11-modules-1

group

inline_ncube_recordlayout

instance

logicalproduct

ncube_recordlayout

organization

physicaldataproduct

physicalinstance

reusable

simpledc20021212

studyunit

tabular_ncube_recordlayout

xml

set of xml schemas to support xhtml

XML SCHEMAS

Open Data Foundation – IZA 2007/11

basic structure organization
Basic Structure/Organization
  • DDI 3.0 is divided into “modules”
  • Each contains a set of related metadata
  • Reusable metadata is divided into schemes
  • Modules reflect the steps of the data lifecycle

Open Data Foundation – IZA 2007/11

ddi 3 0 modules
DDI 3.0 Modules
  • Main modules are:
    • Study Unit (contains a simple study description)
    • Conceptual Component
    • Data Collection (survey instruments, questions, sources)
    • Logical Product (concepts, variables, codes, categories)
    • Physical Storage (describes patterns of storage and physical instances/files)
    • Archive (organizations and processing events)
    • Group (comparing and grouping study units)
    • Comparative (allows for explicit comparisons between grouped studies)
  • See also http://www.ddialliance.org/DDI/ddi3/module-descriptions.html

Open Data Foundation – IZA 2007/11

maintainable schem e s that s with an e not an a
Concept Scheme

Universe Scheme

Question Scheme

Control Construct Scheme

Category Scheme

Code Scheme

Variable Scheme

Packages of reusable metadata maintained by a single agency

Maintainable Schemes (that’s with an ‘e’ not an ‘a’)

Open Data Foundation – IZA 2007/11

ddi 3 0
DDI 3.0
  • Look at schema (Candidate release 2)
  • Look at examples (prototype XML)

Open Data Foundation – IZA 2007/11

slide24

Our Initial Thinking…

The metadata payload from version 2.* DDI was re-organized to cover these areas.

Open Data Foundation – IZA 2007/11

slide25

For later parts of the lifecycle, metadata is reused heavily from earlier

Modules.

Wrapper

The discovery and analysis itself creates data and metadata, re-used in future cycles.

Open Data Foundation – IZA 2007/11

realizations
Realizations
  • Many different organizations and individuals are involved throughout this process
    • This places an emphasis on versioning and exchange between different systems
  • There is potentially a huge amount of metadata reuse throughout an iterative cycle
    • We needed to make the metadata as reusable as possible
  • Every organization acts as an “archive” (that is, a maintainer and disseminator) at some point in the lifecycle
    • When we say “archive” in DDI 3.0, it refers to this function

Open Data Foundation – IZA 2007/11

ddi 3 0 lifecycle model
DDI 3.0 Lifecycle Model

Metadata Reuse

Open Data Foundation – IZA 2007/11

use cases
Use Cases
  • Study design/survey instrumentation
  • Questionnaire generation/data collection and processing
  • Data recoding, aggregation and other processing
  • Data dissemination/discovery
  • Archival ingestion/metadata value-add
  • Question/concept/variable banks
  • DDI for use within a research project
  • Capture of metadata regarding data use
  • Metadata mining for comparison, etc.
  • Generating instruction packages/presentations

Open Data Foundation – IZA 2007/11

study design survey instrumentation
Study Design/Survey Instrumentation
  • This use case concerns how DDI 3.0 can support the design of studies and survey instrumentation
    • Without benefit of a question or concept bank

Open Data Foundation – IZA 2007/11

questionnaire generation data collection and processing
Questionnaire Generation, Data Collection, and Processing
  • This use case concerns how DDI 3.0 can support the creation of various types of questionnaires/CAI, and the collection and processing of raw data into microdata.
  • Algenta working on DDI 3.0 based software

Open Data Foundation – IZA 2007/11

data recoding aggregation etc
Data Recoding, Aggregation, etc.
  • This use case concerns how DDI 3.0 can describe recodes, aggregation, and similar types of data processing.
  • Relevant to both producer and researcher

Open Data Foundation – IZA 2007/11

data dissemination data discovery
Data Dissemination/Data Discovery
  • This use case concerns how DDI 3.0 can support the discovery and dissemination of data.
  • Highly relevant to researchers

Open Data Foundation – IZA 2007/11

archival ingestion and metadata value add
Archival Ingestion and Metadata Value-Add
  • This use case concerns how DDI 3.0 can support the ingest and migration functions of data archives and data libraries.

Open Data Foundation – IZA 2007/11

question concept variable banks
Question/Concept/Variable Banks
  • This use case describes how DDI 3.0 can support question, concept, and variable banks. These are often termed “registries” or “metadata repositories” because they contain only metadata – links to the data are optional, but provide implied comparability. The focus is metadata reuse.
  • Concept classification very important to researchers

Open Data Foundation – IZA 2007/11

ddi for use within a research project
DDI For Use within a Research Project
  • This use case concerns how DDI 3.0 can support various functions within a research project, from the conception of the study through collection and publication of the resulting data.
  • Direct use in RDC

Open Data Foundation – IZA 2007/11

capture of metadata regarding data use
Capture of Metadata Regarding Data Use
  • This use case concerns how DDI 3.0 can capture information about how researchers use data, which can then be added to the overall metadata set about the data sources they have accessed.
  • Data use and user feedback crucial to improve overall quality and future data production (relevance)

Open Data Foundation – IZA 2007/11

metadata mining for comparison etc
Metadata Mining for Comparison, etc.
  • This use case concerns how collections of DDI 3.0 metadata can act as a resource to be explored, providing further insight into the comparability and other features of a collection of data.

Open Data Foundation – IZA 2007/11

generating instruction packages presentations
Generating Instruction Packages/Presentations
  • This use case concerns how DDI 3.0 can support automation around the instruction of students and others.

Open Data Foundation – IZA 2007/11

tools
Tools
  • DDI 1/2.x
    • IHSN Microdata Management Toolkit (http://www.surveynetwork.org/toolkit)
    • Nesstar (http://www.nesstar.com)
    • http://www.ddialliance.org
    • Dextris (http://opendatafoundation.org)
  • DDI 3.0
    • Foundation Tools Platform
    • UKDA DExT
    • DDI 3.0 Use case http://opendatafoundation.org/ddi/use_cases.php
    • Algenta SurveyWiz
    • Dextris (http://opendatafoundation.org)

Open Data Foundation – IZA 2007/11

ddi for rdc
DDI for RDC
  • A small set of DDI 1/2.x tools are available today
    • Users can generate from internal databases, use conversion utilities (see DDI web site) or software like Nesstar Publisher and the IHSN Microdata Management Toolkit
  • DDI 3.0 has a much broader scope and provides both core and advanced functionalities that will require management tools
    • Next generation metadata framework is being build as the standard is begin finalized
    • The DDI Foundation Tools Program is an umbrella for implementers startup toolkit

Open Data Foundation – IZA 2007/11

conclusions
Conclusions
  • The first generation of DDI is suitable for data archives interested in the preservation of metadata and discovery by users
  • DDI 3.0 focus on the entire life cycle of the survey and is suitable for many different uses.
    • More relevant to RDC environment
  • DDI 3.0 calls for coordinated efforts for building relevant tools for producers, archives, researchers and other users

Open Data Foundation – IZA 2007/11