1 / 16

EMS Flat File Interfaces

EMS Flat File Interfaces. 10 October 2007 http://ems.eos.nasa.gov. Objectives. Overview of flat file validation and processing Overview of product and user lookups Implications of revision files. Process Overview. EMS Validation Process. All data/file types validated against ICD

steven-carter
Download Presentation

EMS Flat File Interfaces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EMS Flat FileInterfaces 10 October 2007 http://ems.eos.nasa.gov

  2. Objectives • Overview of flat file validation and processing • Overview of product and user lookups • Implications of revision files

  3. Process Overview

  4. EMS Validation Process • All data/file types validated against ICD • Only valid records ingested into EMS • Detailed report provided for failed records • Email • Apex/HTMLDB Data Provider status tab

  5. EMS Notifications • File auditing • Data File Manifest (DFM) determines what is expected and frequency • Missing files • Unknown files • Reports available • Email • Apex/HTMLDB Data Provider status tab

  6. Users, Meta, & Search Expression • EMS Tables • Base tables only • Data as is from flat file • Used in lookup process for Ingest, Archive, and Distribution files • Revision Files • EMS merges data • No physical delete

  7. Ingest & Archive Data • EMS Tables • Base: data as is from flat file • Summary • Summarized history • Contains product mapping data • Updated Daily • Revision Files • EMS merges data • No physical delete • Archive: deleteFromArchive flag

  8. Distribution Data • Distribution Process (post validation) • Look ups • Users • Get user from user profile data by userID, email, or IP/host • Default: “Anonymous” • Product • Get product from product search data by request path containing search expression • Get product mapping from meta data by product • Default: “Not Provided”

  9. Distribution Data • EMS Tables • Base • contains data as is, plus user profile and product mappings • Summary • Daily summary

  10. Distribution Data • Revision Files • Previously existing daily log deleted • Latest revision file inserted • Latest user profile, product, and search expression mappings applied

  11. Backup

  12. Distribution User Lookup • Prior to performing lookups all IP address are resolved to hostnames • If a valid userID exists in the Distribution Data File and the User Profile Flat File the corresponding userType and affiliation will be assigned and the next record will be evaluated. • If a valid email address exists in the Distribution Data File and the User Profile Flat File the corresponding userType and affiliation will be assigned and the next record will be evaluated. • If a valid hostname or IP address exists in the Distribution Data File and the User Profile Flat File the corresponding userType and affiliation will be assigned and the next record will be evaluated. • In no match occurs the record will be assigned the Public user type and the affiliation will be populated with ‘Unknown’ and the next record will be evaluated.

  13. Data Type Definitions * Required fields – http://ems.eos.nasa.goc/document.....

  14. EMS Naming <YYYYMMDD>_<Provider>_<FileType>_<DataSource>.flt.rev<1-n> Timestamp Extension Root File Name

  15. Acronyms • CCR Change Control Request • DFM Data File Manifest • DP Data Provider • EMS EOSDIS Metrics System • ICD Interface Control Document

More Related