1 / 30

ECS SDP Internal Training

PRODUCTION. Generate. Plan. INSERTION. RETRIEVAL. Deliver. Ingest. Search & Order. Subscribe. Store. External Data Provider. User. ECS SDP Internal Training. Revised July 1999 (Reflects Release 5A). Objectives.

montana-griffith
Download Presentation

ECS SDP Internal Training

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PRODUCTION Generate Plan INSERTION RETRIEVAL Deliver Ingest Search & Order Subscribe Store External Data Provider User ECS SDP Internal Training Revised July 1999 (Reflects Release 5A)

  2. Objectives • Overall objective: Describe ECS structure and function for Science Data Processing (SDP) • Identify subsystems and Computer Software Configuration Items (CSCIs) • Specify major components and functions/processes of CSCIs • Describe role of CSCIs/functions/processes in the context of ECS operational scenarios • ASTER-specific functions (e.g., DAR, expedited data support) • Producing and distributing data products (including media) • Updating QA metadata • On-demand processing workaround • User registration • Landsat data insertion and access

  3. What This Lesson Is (and Is Not) • Is • Brief illustration of ECS high-level structure • Introduction to subsystems that make up ECS at a site • Examination of each subsystem and its Computer Software Configuration Items (CSCIs), with components • Introduction of all system elements and brief description of functions • Background for subsequent scenario-based presentation of system functional flows • Detailed look at system functioning in the context of operational scenarios • Is Not • Full description of overall ECS structure and function • Description of specific individual ECS entities (e.g., SMC) • Software development lesson • Complete description of interfaces and event sequences • Operations training

  4. U.S. Global Change Research Program NASA Earth Science Enterprise Earth Science Enterprise: - Advance scientific understanding of the entire Earth system Earth Observing System Earth Probes & Focused Missions EOS: - Principal element of Earth Science Enterprise - Designed to collect Earth Science Data - 3 Components: space system, ground system, research system. Science Satellites EOSDIS UARS TOPEX TRMM TOMS Other Terra (AM) PM CHEM Landsat-7 Other EDOS EBNET ECS 28 GB/Day Raw Data 460 GB/Day Raw Data EOSDIS EOSDIS: - Provides the ground system for collection and analysis of science data 1600 GB/Day Processed Data Science Data User Science Data User: - Science Researchers Program Overview

  5. Distribution/Access S/C S/C Data Operations Acquisition ECS EDOS Data Data Communications Communications USERS ECS Mission Operations System (EMOS) TDRSS EOS Spacecraft USERS EOS Data and Operations System (EDOS) EOSDIS EOSDIS Backbone Network White Sands External (EBnet) Complex Network USERS Science Data Processing (SDP) X-Band Backup Ground Stations USERS EOSDIS Ground Stations EOSDIS KEY ECS EDOS Data Comm EOSDIS Principal Components

  6. DAACs ASF GSFC LaRC NSIDC EDC JPL SEDAC ORNL Science Data Processing EOSDIS Backbone Network (EbNet) International Partners Version 0 External Networks (NI/NSF) Instrument Support Toolkit PI/TL Facility Affiliated Data Centers/Other Data Centers Science Community Science Computing Facilities Investigators Working Group EOSDIS Data Flow EDOS Flight Dynamics System ECS Mission Operations System (EMOS) Communications and Internetworking Infrastructure Data from other Mission Data Systems (Landsat-7, TRMM, UARS, . . .) Totally supplied by ECS Contains ECS components

  7. USGCRP Earth Science Enterprise EOS EOSDIS ECS Relationship of ECS to Global Change Research WORLDWIDE GLOBAL CHANGE RESEARCH

  8. Communications (CSS) and Internetworking (ISS) User Registration, Order Status, User Profile PUSH Client (CLS) External/ Internal Users Interoper- ability (IOS) Mgmt Support Subsystem MSS Planning Subsystem (PLS) abc def ghi jkl mno Planning Plans and On-demand Requests Data Processing Requests Documents/ Advertisements Search and Access Data Processing Subsystem (DPS) 0110001011 0010110010 1101011101 0110001011 Data Data Mgmt Subsystem (DMS) Data Proc Data Server Subsystem (DSS) Data/Services Collection Ingest Subsystem (INS) DDIST SDSRV Ingest Inven- tory DB Acquire Science Software and Data PULL STMGT External Data Sources and Providers Insert and Retrieve Archive EMOS ECS Context

  9. Subsystems and Functions Science Data Processing Segment (SDPS) • Data Server Subsystem (DSS) • Data storage and management: archive science data (with related insert, search and retrieve functions), archive management, data resource staging • Ingest Subsystem (INS) • Interface with external data providers and transfer data into ECS (with related staging functions and operator interfaces) • Data Processing Subsystem (DPS) • Dispatches and monitors execution of science software • Planning Subsystem (PLS) • Long- and short-term planning of science data processing, and management of production resources

  10. Subsystems and Functions (Cont.) SDPS (Cont.) • Client Subsystem (CLS) • Provides interfaces and access for external users • Data Management Subsystem (DMS) • Enables cross-site data search and retrieval; gateways for interface of ECS with EOSDIS Data Gateway Web Client (Version 0 IMS) protocol • Interoperability Subsystem (IOS) • Advertising Service; support for other subsystems in locating data or DSS services

  11. Subsystems and Functions (Cont.) Communications and System Management Segment (CSMS) • System Management Support Subsystem (MSS) • System maintenance, management, and administration (includes trouble ticketing, baseline and configuration management, fault and performance monitoring, and user account management and order tracking) • Communications Subsystem (CSS) • General system infrastructure functions (includes DCE and network communications, libraries to standardize software mechanisms, application error handling, interfaces to e-mail, file transfer and network file copy functions) • Internetworking Subsystem (ISS) • Networking hardware devices and embedded software NOTE: The ISS is part of the ECS infrastructure and is not addressed in detail in this course.

  12. Communications (CSS) and Internetworking (ISS) User Registration, Order Status, User Profile PUSH Client (CLS) External/ Internal Users Interoper- ability (IOS) Mgmt Support Subsystem MSS Planning Subsystem (PLS) abc def ghi jkl mno Planning Plans and On-demand Requests Data Processing Requests Documents/ Advertisements Search and Access Data Processing Subsystem (DPS) 0110001011 0010110010 1101011101 0110001011 Data Data Mgmt Subsystem (DMS) Data Proc Data Server Subsystem (DSS) Data Server Subsystem (DSS) Data/Services Collection Collection Ingest Subsystem (INS) DDIST DDIST SDSRV SDSRV Ingest Inven- tory DB Inven- tory DB Acquire Science Software and Data PULL STMGT STMGT External Data Sources and Providers Insert and Retrieve Archive EMOS Subsystems and CSCIs: DSS

  13. Data Server Subsystem Collection DDIST SDSRV Inven- tory DB STMGT Subsystems and CSCIs: DSS (Cont.) • Data Server Subsystem (DSS) • Provides capabilities to store, search, retrieve, and distribute earth science and related data • Client-server information transfer is by commands and requests • Uses Universal References to identify ECS entities • GranuleUR: represents a granule in the data server (e.g., as follows) UR:10:DsShESDTUR:UR:15:DsShSciServerUR:13:[GSF:DSSDSRV]:16:SC:MOD10_L2:1411 • DsServerUR: represents a specific running data server application (e.g., ) [GSF:DSSDSRV] • Uses MSS Event services to log system-level events • Interfaces with virtually all ECS subsystems and components • Uses several COTS tools: RogueWave tools and libraries, Sybase relational database, Spatial Query Server, HDF, ODL

  14. SDSRV Inven- tory DB Subsystems and CSCIs: DSS (Cont.) • Science Data Server (SDSRV) CSCI • Provides the ECS with a catalog of Earth Science Data holdings, and the Earth Science Data Type (ESDT) services that operate on the data • Manages and provides user access to data collections through its catalog of metadata and mechanisms to acquire data from the archive • Four major components • Science Data Server - services requests for storage, search, retrieval, and manipulation of science data • HDF EOS Server - provides science data subsetting • Science Data Server GUI - provides operator interface • Sybase/SQS Server - manages catalog (metadata)

  15. EcDsHdfEosServer EcDsSdSrvGui ECS Inventory Database SQS (Sybase) EcDsScienceDataServer Subsystems and CSCIs: DSS (Cont.)SDSRV Architecture and Interfaces

  16. STMGT Subsystems and CSCIs: DSS (Cont.) • Storage Management (STMGT) CSCI • Stores, manages, and retrieves data files on behalf of other science data processing components • Five major components • Archive Server - provides GUI and access to stored data • Staging Monitor - manages the group of data files that have been retrieved from the archive and placed into a cache area on staging disk • Resource Manager - schedules access to shared peripheral devices • Pull Monitor - manages files in the user pull area, deleting them as they are retrieved by users or as their time-out periods expire • Data Base - contains data tables for STMGT devices, cache management, event and log management, requests, and related functions

  17. EcDsStArchiveServer Device Info Request List Staging Data List AMASS EcDsStPrintServer Resource List EcDsStStagingMonitorServer STMGT/DDIST Database Disk Index File SQL (Sybase) EcDsStPullMonitorServer EcDsStStagingDiskServer EcDsSt8MMServer EcDsStD3Server EcDsStCDROMServer EcDsStFtpDisServer EcDsStmgtGui EcDsStIngestFtpServer External Gateways Subsystems and CSCIs: DSS (Cont.)STMGT Architecture and Interfaces

  18. Collection DDIST Subsystems and CSCIs: DSS (Cont.) • Data Distribution (DDIST) CSCI • Formats and distributes data to users, either electronically or on physical media (e.g., 8mm tape cartridges) • Directs STMGT to place data in working storage, to copy data to tape, or to push data as required via FTP • Sends e-mail notifications • Three major components • Data Distribution Server - provides control and coordination for data distribution through request processing • Data Distribution GUI - allows operations staff to initiate, track, and manipulate distribution requests • Data Base - contains the request list; updates and provides the request configuration

  19. STMGT/DDIST Database SQL (Sybase) EcDsDistributionServer EcDsDdistGui Subsystems and CSCIs: DSS (Cont.)DDIST Architecture and Interfaces

  20. Data Distribution EcDsDistributionServer Communications (CSS) and Internetworking (ISS) User Registration, Order Status, User Profile Science Data Server EcDsScienceDataServer Archive Server EcDsStArchiveServer EcDsStmgtGui Staging Monitor Process EcDsStStagingMonitorServer EcDsStStagingDiskServer Data Distribution GUI EcDsDdistGui PUSH HDF EOS Server EcDsHdfEosServer Client (CLS) External/ Internal Users Interoper- ability (IOS) Mgmt Support Subsystem MSS Data Base Sybase Server (COTS) Planning Subsystem (PLS) Resource Manager Process EcDsSt8MMServer EcDsStD3Server EcDsStIngestFtpServer EcDsStFtpDisServer EcDsStCDROMServer EcDsStPrintServer abc def ghi jkl mno Science Data Server GUI EcDsSdsrvGui Planning Plans and On-demand Requests Data Processing Requests Documents/ Advertisements Data Distribution (DDIST) CSCI Search and Access Data Processing Subsystem (DPS) 0110001011 0010110010 1101011101 Sybase/SQS Sybase Server (COTS) 0110001011 Data Data Mgmt Subsystem (DMS) Data Proc Pull Monitor Process EcDsStPullMonitorServer Data Server Subsystem (DSS) Data Server Subsystem (DSS) Data/Services Science Data Server (SDSRV) CSCI Data Base Sybase Server (COTS) Collection Collection Ingest Subsystem (INS) DDIST DDIST SDSRV SDSRV Storage Management (STMGT) CSCI Ingest Inven- tory DB Inven- tory DB Acquire Science Software and Data PULL STMGT STMGT External Data Sources and Providers Insert and Retrieve Archive EMOS Subsystems and CSCIs: DSS (Cont.)

  21. Ingest Subsystem Ingest Subsystems and CSCIs: INS • Ingest Subsystem (INS) • Transfer of data into ECS (SDPS repositories) in accordance with approved ICDs • Supports varied data formats and structures • Ingest Client: A set of ingest software configured for requirements of a specific situation • Ingest clients perform data preprocessing, metadata extraction, and metadata validation on incoming data • Data staged to one of two areas • Level 0 (L0) data from ongoing missions, and EDOS ancillary data, staged to INS working storage area • Non-L0 data (e.g., non-EDOS ancillary data, L1A-L4 data) staged directly to DSS working storage area • Uses several COTS tools: RogueWave class libraries, Sybase relational database, Tivoli Client, DCE Client, DCE Driver, MSAccess, HP OpenView Client

  22. Subsystems and CSCIs: INS (Cont.) • Ingest (INGST) CSCI • Acquires data by various methods and transfers the data into ECS • Automated transfer: in response to notification from the data provider, Ingest transfers the data from a specified network location • Polling: transfer of data from predetermined network locations which Ingest periodically checks for new data • With Delivery Record • Without Delivery Record • Media: reading data from tapes • Stores and manages request information • Provides for data preprocessing and insertion

  23. Subsystems and CSCIs: INS (Cont.) • Ingest (INGST) CSCI (Cont.) • Seven major components • Automated Network Ingest Interface - provides basic capability to ingest data electronically from an external source • Polling Ingest Client Interface - creates polling request, detects new files in a specified external location, creates and submits ingest request • Media Ingest Interface - provides operators ability to perform ingest from physical media • Cross-Mode Ingest Interface - provides an E-mail gateway server to receive distribution notifications and store them as files in a location that can be polled to create delivery records • Ingest Request Manager - manages ingest request traffic and processing • Ingest Granule Server - provides services for required preprocessing of data and subsequent insertion into Data Server • Ingest Database - stores and provides access to Ingest Subsystem internal data (e.g., History Logs)

  24. Local Disk External Data Providers Ingest Database SQL (Sybase) DAAC Ingest Technician CSS Gateway EcInAuto EcInPolling EcInGUI EcInEmailGWServer EcInGran EcInReqMgr Subsystems and CSCIs: INS (Cont.)Architecture and Interfaces

  25. Communications (CSS) and Internetworking (ISS) User Registration, Order Status, User Profile Media Ingest Interface Process EcInGUI Automated Network Ingest Interface Process EcInAuto PUSH Client (CLS) External/ Internal Users Polling Ingest Client Interface Process EcInPolling Ingest Request Manager Process EcInReqMgr Interoper- ability (IOS) Mgmt Support Subsystem MSS Planning Subsystem (PLS) abc def ghi jkl mno Planning Plans and On-demand Requests Data Processing Requests Documents/ Advertisements Ingest Granule Server Process EcInGran Cross-Mode Ingest Interface EcInEmailGWServer Search and Access Data Processing Subsystem (DPS) 0110001011 0010110010 1101011101 0110001011 Data Data Mgmt Subsystem (DMS) Data Proc Ingest Database Sybase Server (COTS) Data Server Subsystem (DSS) Ingest (INGST) CSCI Data/Services Collection Ingest Subsystem (INS) Ingest Subsystem (INS) DDIST SDSRV Ingest Ingest Inven- tory DB Acquire Science Software and Data PULL STMGT External Data Sources and Providers Insert and Retrieve Archive EMOS Subsystems and CSCIs: INS (Cont.)

  26. Client Subsystems and CSCIs: CLS • Client Subsystem (CLS) • User access to ECS services and data, and other systems interoperable with ECS (e.g., Version 0) • Search and retrieval of data are performed by the EOS Data Gateway (Version 0 Web Client) • Includes applications programs accessible through user interfaces • User Registration Tool (URT) • EOSView • Java Data Acquisition Request (DAR) Tool • Uses several COTS tools: Netscape Navigator, Netscape Enterprise Server, XVT (widget set and development tool for EOSView), Interactive Data Language (IDL) (used in EOSView visualization features), and Rogue Wave Tools.h++ (libraries provide strings and collections to the Java DAR Tool and User Registration Tool)

  27. Subsystems and CSCIs: CLS (Cont.) • Workbench (WKBCH) CSCI • Includes applications and libraries for access to ECS data and services • Release 5 includes 3 tools • URT (HTML-based) • EOSView (X/Motif-based) • Java DAR Tool (Java/HTML-based) • Since the CLS contains no hardware CIs, the DMS hardware (Interface Servers) provides the processing and storage for the WKBCH software • In addition, the User Profile Gateway Server is used to provide user profile information to the EOS Data Gateway for ECS users

  28. Science/DAAC User EOS Data Gateway Web Client User Site User Profile File (profile.odl) DAAC Site (Design controlled under separate contract) EcClDtUser ProfileGateway CLS Subsystems and CSCIs: CLS (Cont.) Search & Order Tool Interfaces

  29. Netscape Enterprise Server Web Browser Science/DAAC User HDF File EcClWbUr EcClWbUrConfirm EcClWbUrSubmit EcClWbJdt WKBCH EOSView CLS Subsystems and CSCIs: CLS (Cont.)WKBCH Architecture and Interfaces

  30. Communications (CSS) and Internetworking (ISS) User Registration, Order Status, User Profile User Registration Tool (URT) Scripts EcClWbUr EcClWbUrConfirm EcClWbUrSubmit PUSH Client (CLS) External/ Internal Users Client (CLS) External/ Internal Users Interoper- ability (IOS) Mgmt Support Subsystem MSS User Profile Gateway EcClDtUserProfileGateway Planning Subsystem (PLS) abc def ghi jkl mno Planning Plans and On-demand Requests Data Processing Requests Documents/ Advertisements Search and Access HDF Product Viewing Tool EOSView Data Processing Subsystem (DPS) 0110001011 0010110010 1101011101 0110001011 Data Data Mgmt Subsystem (DMS) Data Proc Java DAR Tool EcClWbJdt Data Server Subsystem (DSS) Data/Services Collection Ingest Subsystem (INS) DDIST Workbench (WKBCH) CSCI SDSRV Ingest Inven- tory DB Acquire Science Software and Data PULL STMGT External Data Sources and Providers Insert and Retrieve Archive EMOS Subsystems and CSCIs: CLS (Cont.)

More Related