Structural and Technical Issues with DDI 3.1 and Suggested Improvements

1. Structural and Technical Issues with DDI 3.1 and Suggested Improvements Arofan Gregory J Gager Metadata Technology NA

2. Background • Metadata Technology has been working on many different implementations of DDI 3.1 • Specific implementations for different organizations • Generalized approach to produce useful open source

3. Issues with DDI Lifecycle (1) • Too much coverage of corner cases • Makes the 80% case too difficult to implement • Changes across versions have too much impact • DDI 3.2 was not a minor release in real terms • DDI 3.2 is not needed for most typical cases and we doubt it will be widely adopted outside of a few organizations which need specific features • Some important constructs are too broadly defined • Schemes can represent many different things • Studies can represent many different things

4. Issues with DDI Lifecycle (2) • Schemes are not lists of like items • But they should be! • In many cases, there is more than one way to do a single thing • Is this really necessary? • It destroys interoperability in practical terms • Why are packaging elements such as modules maintainable? • These things don’t exist in real systems

5. Issues with DDI Lifecycle (3) • Versioning is too strict • For many cases, changes only get time-stamped because the specified system can’t be supported by actual systems • Not enough guidance on what constitutes a versionable change– this could be defined in a useful way • Modularity isn’t done correctly • Namespaces do not align with actual use!

6. Issues with DDI Lifecycle (4) • There is no web-services orientation • DDI is very often deployed as Web Services • But there are no standard queries, functions, or interfaces • Registry interfaces are not specified • Repository interfaces are not specified

7. Proposed Improvements – OO Model/Schemas • When we move to a model-driven process, there are some important things we should not lose • The object-oriented nature of the XML schemas is a powerful, useful tool, which could be captured in a pure conceptual model • SDMX is a good example of how this can be done • Makes the model and schemas much easier to produce and maintain

8. Proposed Improvements - Modularity • The “resusable” module should instead be a true “core” module • Not just any re-used element • Modules should be functionally oriented • Describe a survey • Describe a simple data set • Etc. • Models should import the core and focus on a specific function by adding the most-needed elements only • Specific models could be different XML documents • Easier to understand • Easier to implement • More realistic versioning mechanisms

9. Proposed Improvements - Extensions • A good extension mechanism should be provided • Only one! • Corner-cases should be handled through extensions • Not through an overly complex “standard” model • This approach requires a clear definition of intended scope for the standard

10. Suggested Improvements – Schemes and Modules • The things which are “schemes” today should be split apart and specifically defined • Schemes for processing • Schemes for archiving • Etc. • The packaging elements should not all be maintainable • Fewer levels of maintainables above the item level • As few as possible • Look at SDMX!

11. Suggested Improvements - SOA • Develop queries for typical functions • Describe a standard RESTful interface • Describe standard SOAP function • Include both registry and repository interfaces

12. Suggested Improvements – A Comment on Process Modelling • DDI should not reproduce the functionality of existing, well-supported standard for process modelling • BPMN • BPEL • Others • DDI should focus on doing what it does better • Focus on data • Danger is having too broad a focus which provides less-well-developed solutions