Exchange Design Best Practices

Exchange DesignBest Practices Tools for Successful Flow Design and Implementation

Flow Design & Implementation Life Cycle • Planning • Flow architecture design • Determine data elements • Draft Schema and submit for initial review • Develop flow components (sender/receiver) • Implement and test • Document and submit for final review and publishing

Planning Phase • Determine goals of the exchange • Often a regulatory reporting requirement involved (Submit) • Consider data publishing model • Expose queries • Make receiver “Come and get it” • Still satisfies regulatory requirements • Consider other uses for the data – make the exchange flexible! • Ad hoc uses of data (human) • Support warehousing by any partner (machine)

Flow Architecture Design Phase • Determine how to chunk data • Schema and Data Serviceswork hand in hand • Compartmentalize Schema by data module • Enables returning portions of data • GetFacilityByName(“ACME”) • GetFacilityContactsByFacilityID(“1234”)

Flow Architecture Design Phase • Design feedback mechanisms and fault condition behavior • Many teams only considered the “happy flow” • Rejected files and partial rejection behavior should be considered and documented • “GetStatus” responses should be explicitly defined for each stage of processing

Flow Architecture Design Phase • Determine how to handle Transactional Processing • Ideal scenario – Avoid it! • Data provider should need no prior knowledge of receiver’s data • Receiver should know what data to request and how to consume the returned data • MUCH easier for data providers to implement • If it can’t be avoided… • Use Header “Payload” element to track “Insert/Update” and “Delete” • Last resort: Use transaction codes in schema

Determine Data Elements • DON’T design the schema by mirroring an existing database model • Hierarchal structure of schema is best for maximizing interoperability • Relationships expressed through nesting, not keys • Key/Keyref requires data provider to do more elaborate data staging • Goal is to make the data abstract • That said, evaluate each data source for candidate elements

Schema Design and Draft Review • Schema design: How rigid should it be? Flexible: • no explicit field lengths, fewer required fields, no Key/Keyref, embedded code lists • Use the Shared Schema Components!!! Rigid: • Shorter shelf life (i.e. code lists change) • Usually done to be target system-centric • Limits likelihood that schema will be reused by other flows or matches data standards Flexible Rigid

Schema Design and Draft Review • Compromise! • Keep schema loose (flexible) • Use schematron to enforce target-specific rules • EPA CDX hosts a schematron validation • Few resources to help with Schematron  • More powerful than good old schema validation • Schematron rules can be modified if business rules change without revising the schema!

Schema Design and Draft Review • Using the Shared Schema Components

Schema Design and Draft Review IMPORTANT THING #1 • Follow the design guidance! • Review the guidance on the EN web site! • If you have questions, get answers! IMPORTANT THING #2 • Submit for Initial review! • Exchange Network offers “early engagement” services to help uncover issues with your schema and exchange design before sinking significant resources in flow processing code! • Major issues have been identified every time! We are here to help!

Develop Flow Components • Build the data staging tables • Good practice: stage the data in a data model similar to the schema format • Create SQL scripts for staging tables and publish with the exchange • Build the flow composition/decomposition components • Use OO design to build parsing routines • Maximizes the likelihood for reuse! • All flow deliverables (including code) are part of the public domain

Implementation and Testing • Testing will expose problems • Document issues as they arise • Other implementers will encounter the same issues • Performance issues with large files • Test iteratively • Be prepared to revise schema and/or data services if needed • Test between multiple partners • Test end-to-end, including feedback

Submit Flow Package for Review • Prepare flow documentation package • Complete and clean up the FCD, DET, and schemas • Create sample XML instance documents • Finalize Schema Conformance Report • Review EN guidance before submitting • Required/Optional materials • Proper schema packaging • Index.xsd • Folder structure • Submit for review – (2-6 week turn around)

Schema Design Rules and Guidance ***Use other published flows as a reference

Pop Quiz Question: A developer is not sure how to design some aspect of a schema or flow. What should he/she do? Answer: • Ignore the problem. It will go away. • Just guess and move on. • Obfuscate the issue with confusing language in the flow documentation. • Review the guidance documents. If no answer if found, ask the NTG for assistance.

Schema Design Do’s DO: • Read and follow the Design Rules and Conventions (DRC) v1.0 and v1.1 • Follow Naming Rules (Schema elements and types, Namespace, and Data Services) • Follow versioning rules • Use the Shared Schema Components in your schema • Follow schema file segmentation guidelines • Ask questions • Submit your draft schema for review

Flow Configuration Document Do’s DO: • Thoroughly document all data services (parameters, order, required/optional, return schema) • Thoroughly document payload structure (use of Header, zip files, payload operations, arrays of NodeDocuments) • Assume the reader is not familiar with your flow • Use the FCD template and other FCDs for reference • If the Header is used, thoroughly document how and when it is used

Flow Configuration Document Don’ts DON’T: • Include non-flow specific stuff (i.e. how to ping, what the header is) • Forget to remove comments, track changes or other editing artifacts

The Moral of the Story • Take advantage of flow development assistance services offered by the NTG! • Read, understand, and live the guidance! • Sometimes there are no easy answers... Ask questions! Buffer your decisions!

Exchange Design Best Practices