espds nde build 4 and 5 design description build content review bcr n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
ESPDS NDE Build 4 and 5 Design Description Build Content Review (BCR) PowerPoint Presentation
Download Presentation
ESPDS NDE Build 4 and 5 Design Description Build Content Review (BCR)

Loading in 2 Seconds...

play fullscreen
1 / 233

ESPDS NDE Build 4 and 5 Design Description Build Content Review (BCR) - PowerPoint PPT Presentation


  • 184 Views
  • Uploaded on

ESPDS NDE Build 4 and 5 Design Description Build Content Review (BCR). NDA-SE13-DOC-1.0 December 13, 2011 Meet me 1-866-762-5577 pc 173812# Web Ex: https://mmancusa.webex.com/mmancusa/j.php?ED=175042952&UID=493176402&PW=NNmY4ZjMwNjJk&RT=MiMxMQ%3D%3D. Introductions. Government Team

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

ESPDS NDE Build 4 and 5 Design Description Build Content Review (BCR)


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
    Presentation Transcript
    espds nde build 4 and 5 design description build content review bcr

    ESPDS NDEBuild 4 and 5 Design DescriptionBuild Content Review (BCR)

    NDA-SE13-DOC-1.0

    December 13, 2011

    Meet me 1-866-762-5577 pc 173812#

    Web Ex: https://mmancusa.webex.com/mmancusa/j.php?ED=175042952&UID=493176402&PW=NNmY4ZjMwNjJk&RT=MiMxMQ%3D%3D

    introductions
    Introductions
    • Government Team
      • GeofGoodrum – NDE Architect
      • Tom Schott – Polar Products Manager
    • Contractor Team
      • Dan Beall – Program Manager
      • Richard Sikorski – Solers Project Manager
      • Peter MacHarrie – Chief Architect
      • Lou Fenichel – Software Development Lead
    purpose of this bcr
    Purpose of this BCR
    • Why are we here?
      • To obtain approval to create the Customer Services and Monitoring and Control Builds that complete NDE’s Data Handling System (DHS)
    • What are we going to see?
      • Functional descriptions, graphical user interface (GUI) mockups, operational scenarios, and programmatic details
    agenda
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    background
    Background
    • BCR Focus
    • Who uses NDE?
    • What does Builds 4/5 Add?
      • What does it look like?
    background1
    Background
    • Initial NPP Launch Support
      • Build 1: Ingest
      • Build 2: Product Generation
      • Build 3: Distribution
      • Supporting Software/Configurations
    • Builds 4 and 5
      • Deliver concurrently
      • Contain Customer Services and Monitoring and Control capabilities and Build 1,2, & 3 GUIs
      • Complete the NDE DHS
    background2
    Background
    • Paradigm shift – NDE is a FRAMEWORK for integrating multiple product systems into a single Enterprise
    • Major Activities
      • Product System Integration into NDE (NUCAPS, MiRS, etc.)
      • Configurable Ingest Data Providers (NPP, DDS, JPSS, etc.)
      • Customer Registration and configurable Distribution across multiple Product Systems
      • Operational Monitoring and Control across multiple Product Systems
    bcr focus
    BCR Focus
    • What does NDE do for the Customer?
      • Gets the customer the products they want in a highly reliable fashion
    • What does NDE do for OSPO?
      • Makes providing this data from NPP & JPSS possible, efficient, accountable, and reliable
    who uses nde1
    Who uses NDE?

    Sys

    Admins

    Help Desk

    NDE

    TALs

    Customers

    PALs

    Key:

    OSPO

    Person to Person

    Person to Machine

    what does build 4 5 add
    What Does Build 4/5 Add?
    • A portal for Customers (Customer Services)
    • An internal portal for TALs , PALs and Help Desk (Monitor & Control)
    • A Data Warehouse for statistics and reporting
    • Metrics gathering and GUIs for Build 1, 2, and 3
    what does it look like
    What Does It Look Like?
    • Internal users only (i.e. not customers)
    • VPN access via Firefox (if outside of NSOF)
    • Will be tested on IE8+ and Firefox3.6+

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    menu contents
    Menu Contents

    System

    Product Management

    Ingest

    Product Generation

    Distribution

    Reports

    generic status summary table
    Generic Status Summary Table

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Generic Table

    Z

    A

    A

    Z

    of 4

    1

    generic status summary table1
    Generic Status Summary Table

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Generic Table

    Z

    Z

    A

    A

    A

    A

    Z

    Z

    Enter filter text…

    Enter filter value …

    Enter filter value …

    Enter filter value …

    of 4

    1

    sample list and view details
    Sample List and View Details

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Sample Details Viewer Page

    Selected fields are editable

    Save edited values

    Or

    reset to original values

    Reset

    Save

    agenda1
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    customer services
    Customer Services
    • Register Customer
    • Distribute
    • Inform
    register customer
    Register Customer
    • Customer Registration
      • Follows the ESPC CCB and CCR process for approving customer access
      • After access is approved, Technical Area Lead (TAL):
        • Configures a Customer Host for ftps push, if needed
        • Enters Customer Profile and Points of Contact
      • Customer is notified of activation and is required to change password
    register customer1
    Register Customer

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Customer Registration Request

    Basic Information

    Point of Contact List

    Customer ID:

    Password:

    Confirm Password:

    Trusted:

    Approval Threshold:

    • Daytime Operator: nco_operator@noaa.gov remove
    • Nighttime Operator: nco_operator_night@noaa.gov remove

    nco_ops

    ***********

    ***********

    100

    GBs/Day

    Role:

    Email:

    Alt Email:

    Organization:

    Phone:

    Fax:

    Primary POC:

    Trusted Users are not affected by Data Denial

    Lead Operator

    nco_lead_ops@noaa.gov

    NCO

    (301) 555-5555

    (301) 555-5556

    Add to List

    Reset

    Submit

    Cancel

    register customer2
    Register Customer

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Customer Profiles

    Reset

    Save

    Add

    distribute
    Distribute
    • Standing Subscription Requests
      • Customer requests data from PAL/Customer Liaison
        • Define functional characteristics of the data flow (products, data volume, latency & timeliness targets)
      • DDAM (Data Distribution Accounts Manager)
        • Approves customer request
        • Initiates ESPC approval process
        • Defines the interface (SLA, External Data Host, firewall rules, username/authentication)
        • Defines the subscription (protocol, push/pull, notification, etc)
        • Verifies testing in NDE POP Mode
      • ESPC Approval to promote to OPS
    • Authorized User (NOT Customer) enters the data
    distribute1
    Distribute
    • Ad hoc Subscription Requests
      • Customer must be registered
      • Customer and Customer Liaison work out subscription details
      • Subscription includes an end date
      • ESPC approval not required, preapproved up to preset threshold
    • Authorized User (NOT Customer) enters the data
    • Retroactive Delivery Supported
      • Subscription can be backdated and those products still available will be sent
    add customer push host
    Add Customer Push Host

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Add Customer Push Host

    Basic Information

    Host name:

    Address:

    Access Type:

    Description:

    Login ID:

    Password:

    Confirm Password:

    Max Connections:

    nde1.star.nesdis.noaa.gov

    140.90.47.10

    FTP

    STAR Test Machine

    st@rU$er

    ***********

    ***********

    5

    Reset

    Submit

    Cancel

    distribute customer data guis
    Distribute – Customer Data GUIs

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Customer Push Hosts

    Reset

    Save

    Add

    distribute customer data guis1
    Distribute – Customer Data GUIs

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Customer Subscription Request

    Basic Information

    Additional Parameters

    Customer:

    Product:

    Type:

    Primary Host:

    Primary Dir:

    Secondary Host:

    Secondary Dir:

    Start date time:

    End date time:

    Size:

    Compression:

    Checksum:

    Obs Latency Target:

    Sys Latency Target:

    Do Not Deliver:

    Spatial Qualifier:

    Other filters:

    Estimated files/day:

    Estimated xfer/day:

    Estimated MB/day:

    28-OCT-11 05:48:00.00000 AM

    --

    Priority:

    /tmp/primary_pushdir

    secondary.host.external.cust

    primary.host.external.cust

    PUSH

    ATMS_BUFR

    crc32

    nco_ops

    High

    Medium

    zip

    Email

    files

    120

    Minutes

    30

    Minutes

    /tmp/secondary_pushdir

    Minutes after Observation

    480

    Notifications

    (50.7834,-94.5339); (21.0307,-63.948)

    Type:

    Frequency:

    Address:

    Delivery options:

    +

    fqs:N_Percent_Missing_Data:range:0,50;

    every

    50

    1200

    Annie.user@noaa.gov

    1200

    1500

    Reset

    Save

    Cancel

    distribute customer data guis2
    Distribute – Customer Data GUIs

    Geospatial filters

    Gazetteer AOI

    NDE_OPS

    NDE Data Handling System

    Polar Map Toggle

    Annie User | Help | Logout

    +

    Customer Subscription Request

    Basic Information

    Additional Parameters

    Customer:

    Product:

    Type:

    Primary Host:

    Primary Dir:

    Secondary Host:

    Secondary Dir:

    Start date time:

    End date time:

    Size:

    Compression:

    Checksum:

    Observation Latency Target:

    System Latency Target:

    Do Not Deliver After:

    Spatial Qualifier:

    Other filters:

    Estimated files/day:

    Estimated xfer/day:

    Estimated MB/day:

    28-OCT-11 05:48:00.00000 AM

    --

    Priority:

    /tmp/primary_pushdir

    Eastern United States

    secondary.host.external.cust

    crc32

    nco_ops

    ATMS_BUFR

    PUSH

    primary.host.external.cust

    High

    zip

    Medium

    Email

    files

    120

    Minutes

    30

    Minutes

    /tmp/secondary_pushdir

    Minutes after Observation

    480

    Notifications

    (50.7834,-94.5339); (21.0307,-63.948)

    Type:

    Frequency:

    Address:

    Delivery options:

    +

    fqs:N_Percent_Missing_Data:range:0,50;

    every

    50

    1200

    Top-left: (50.7834,-94.5339); Bottom-right: (21.0307,-63.948)

    Cancel

    Done

    Annie.user@noaa.gov

    1200

    1500

    Reset

    Save

    Cancel

    distribute customer data guis3
    Distribute – Customer Data GUIs

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Customer Subscription Request

    Basic Information

    Additional Parameters

    Customer:

    Product:

    Type:

    Primary Host:

    Primary Dir:

    Secondary Host:

    Secondary Dir:

    Start date time:

    End date time:

    Size:

    Compression:

    Checksum:

    Observation Latency Target:

    System Latency Target:

    Do Not Deliver After:

    Spatial Qualifier:

    Other filters:

    Estimated files/day:

    Estimated xfer/day:

    Estimated MB/day:

    +

    28-OCT-11 05:48:00.00000 AM

    Add filters

    --

    Current Filters:

    Priority:

    fqs:N_Percent_Missing_Data:range:0,50; remove

    fm:DayNightFlag:list:Day; remove

    /tmp/primary_pushdir

    secondary.host.external.cust

    ATMS_BUFR

    PUSH

    primary.host.external.cust

    crc32

    High

    nco_ops

    zip

    Medium

    Email

    files

    120

    Minutes

    DayNightFlag

    fm

    list

    30

    Minutes

    /tmp/secondary_pushdir

    +

    Minutes after Observation

    480

    Notifications

    (50.7834,-94.5339); (21.0307,-63.948)

    Done

    Type:

    Frequency:

    Address:

    Delivery options:

    +

    fqs:N_Percent_Missing_Data:range:0,50;

    every

    50

    1200

    Annie.user@noaa.gov

    1200

    1500

    Reset

    Save

    Cancel

    distribute customer data guis4
    Distribute – Customer Data GUIs

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Subscription Definitions

    Reset

    Save

    Add

    distribute customer data guis5
    Distribute – Customer Data GUIs

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Subscription Definitions

    Reset

    Save

    Add

    inform
    Inform
    • Provides the Customer the ability to check on the status of their data subscriptions
      • System level status
      • System/Customer Notification
      • Customer specific subscription status
      • Details of delivered and in process product deliveries
    • Customer Portal is in the public DMZ
    • User authentication is managed via https
    • Portal compliant with NOAA web policies identified at: http://www.cio.noaa.gov/Policy_Programs/webpolicies.html
    nde customer portal login
    NDE Customer Portal Login

    NDE Customer Portal

    Portal Login

    Customer ID

    Password

    Log In

    welcome page nominal ops
    Welcome Page (Nominal Ops)

    NDE Customer Portal

    Click here to update password

    Some User | Log Out | Help

    Suspended Subscriptions

    NDE Status: Nominal

    View All

    View All

    View All

    View All

    View All

    Delivery Status

    Failed Deliveries

    Late Deliveries

    Late Deliveries

    Delivery Notifications

    Failed Delivery Notifications

    Host Status

    Down Hosts

    customer password change
    Customer Password Change

    NDE Customer Portal

    Some User | Log Out | Help

    Inactive Subscriptions

    NDE Status: Nominal

    View All

    View All

    View All

    View All

    View All

    Delivery Status

    Failed Deliveries

    +

    Update Password

    Late Deliveries

    Current Password:

    New Password:

    Confirm New Password:

    *************

    Late Deliveries

    *************

    *************

    Delivery Notifications

    Cancel

    Submit

    Failed Delivery Notificati

    Host Status

    Down Hosts

    all subscription view
    All Subscription View

    NDE Customer Portal

    Some User | Log Out | Help

    NDE Status: Nominal

    Subscriptions

    Back

    Delivery Status

    Late Deliveries

    Delivery Notifications

    Host Status

    product drill down
    Product Drill Down

    NDE Customer Portal

    Some User | Log Out | Help

    NDE Status: Nominal

    Subscriptions

    Back

    +

    Product Description

    Delivery Status

    Late Deliveries

    Delivery Notifications

    Host Status

    product drill down1
    Product Drill Down

    NDE Customer Portal

    Some User | Log Out | Help

    NDE Status: Nominal

    Subscriptions

    Back

    +

    Product Description

    Delivery Status

    Late Deliveries

    Delivery Notifications

    Host Status

    welcome page degraded ops
    Welcome Page (Degraded Ops)

    NDE Customer Portal

    Some User | Log Out | Help

    ID: 1 Date: 2011-12-25 12:00:00 Title: Customer Notice Description: Unexpected Hardware Maintenance has resulted in one available PGS Processor node. Many subscriptions have been suspended to accommodate the outage. Call the ESPC Help Desk (301-817-9999) for any urgent matters.

    NDE Status: Degraded

    View All

    View All

    View All

    Delivery Status

    SuspendedSubscriptions

    Late Deliveries

    Failed Deliveries

    Delivery Notifications

    Late Deliveries

    Host Status

    all distribution jobs view
    All Distribution Jobs View

    NDE Customer Portal

    Some User | Log Out | Help

    Deliveries

    NDE Status: Nominal

    Back

    Delivery Status

    Late Deliveries

    Delivery Notifications

    Host Status

    all notification jobs view
    All Notification Jobs View

    NDE Customer Portal

    Some User | Log Out | Help

    Delivery Notifications

    NDE Status: Nominal

    Back

    Delivery Status

    Late Deliveries

    Delivery Notifications

    Host Status

    customer push hosts view
    Customer Push Hosts View

    NDE Customer Portal

    Some User | Log Out | Help

    Hosts

    NDE Status: Nominal

    Back

    Delivery Status

    Late Deliveries

    Delivery Notifications

    Host Status

    agenda2
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    product management
    Product Management
    • Product Management encompasses
      • Products received from External Data Providers
      • Products generated in-house – NOAA Unique Products (NUPs) and Tailored Products (TPs)
    • Products are specified via CM-controlled XML files, tested extensively, and propagated to Operations via Patch
    • NDE Sustainment performs all Product Management – DPA and NALI functionality
    product management1
    Product Management
    • XML File Development and CM Process
    • Product System Integration
      • Ingest – every Product has an XML File
        • Define external data provider if required
      • Product Generation (NUPs and TPs)
        • Algorithms and Production Rules
      • Operational GUIs
    • Product Quality Monitoring
      • Gap Check/World Wind
      • Reconciliations
      • Product Quality Summary Report
    nup tp via dap transition to ops
    NUP/TP via DAP Transition to Ops

    STAR

    Integration (SADIE, IPT)

    Functional Test (ALGTEST, Test)

    Performance Test (POP, Test)

    CCB (Approve Promotion to OPS)

    XML Files

    Install into OPS

    NDE Patch

    DAP

    • Package contains all SPSRB documentation (build instructions, release notes)
    • Patch is created from ‘built’ software
    nde patch contents for product sys
    NDE Patch Contents for Product Sys

    DAP

    Development

    Test

    OPS

    CCB

    Approval

    NDE Patch

    XML

    Authoring Tool

    NDE Patch

    XML File

    Validation

    Install

    Install

    DB

    DB

    DB

    PGS Nodes

    PGS Nodes

    • Product Description tables
    • Algorithm/Production Rule tables
    • External Product Data Source tables
    product management guis
    Product Management GUIs
    • Product Management GUIs are primarily read-only
    • Provide limited capability to alter selected operational attributes
    • Recommend modification access for TALs only
    • Read only access is available to all roles (does not include customers)
    product management2
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Product Data Source Hosts

    Reset

    Save

    Add

    product management3
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Product Description

    Reset

    Save

    product management4
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Product Description

    Reset

    Save

    product management5
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Product Description

    Reset

    Save

    product management6
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Algorithm Description

    Reset

    Save

    product management7
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Algorithm Description

    Reset

    Save

    product management8
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Algorithm to Node

    Algorithm

    Node

    Reset

    Save

    product management9
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Node to Algorithm

    Algorithm

    Node

    Reset

    Save

    product management10
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Production Rules

    Reset

    Save

    product management11
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Production Rules

    Reset

    Save

    product management12
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Production Rules

    Reset

    Save

    enterprise measures
    Enterprise Measures

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    All Measurements

    Links to Product Content Detail

    of 4

    1

    enterprise measures1
    Enterprise Measures

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Product Content Detail

    product quality management
    Product Quality Management
    • Gap Check
      • To identify missing granule coverage
      • Check Chart
      • World Wind
    • Reconciliations
      • IDPS
      • CLASS
    • Product Quality Summary Report
    gap check chart
    Gap Check Chart

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Product Coverage

    of 96

    1

    nasa world wind gap checker
    NASA World Wind Gap Checker

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    VIIRS M15 EDR CONUS Production Rule

    Spatial Area

    reconciliations
    Reconciliations
    • IDPS
      • Data Delivery Report files received hourly (per NPOESS to NOAA ICD)
      • Cataloged is checked for each file
      • Warning Alerts are generated
    • CLASS
      • Daily File Reconciliation Report is generated (per NDE to CLASS ICD)
      • CLASS personnel provides a notification to ESPC Operators for ‘missing’ files
      • ESPC Operations takes action based on the number/type of missing files
    product management13
    Product Management

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    • Product Quality Summary
    agenda3
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    operations
    Operations
    • Roles
      • Help Desk (Operator) – Read only access to system status
      • TALs (NDE Admin/Engineers)
        • Start and Stop
        • Modify System Element Configuration
    • Process
      • Help Desk logs Trouble Ticket (TT) and posts in Ops Log
        • In a critical situation also calls TAL
      • TALs able to change system element configuration without direct CCB approval for emergencies – otherwise approval required
    • All Changes via GUI logged by time and User ID
    operations1
    Operations
    • Starting the System
    • Is it working?
    • Operators Log
    • Scenarios - Is something wrong?
      • Ingest
      • Product Generation
      • Distribution
      • Infrastructure
    • Quiesce / Stop the System
    • Toggle Data Denial
    nde dashboard
    NDE Dashboard

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Product Generation

    Distribution

    Ingest

    Clean-up

    --m ago

    41.6/86.2 TBs

    SAN

    48% used

    Stats Period

    most recent

    0

    0

    0

    0

    0

    0

    Backlog

    Throughput

    Backlog

    Throughput

    Backlog

    Throughput

    1 hr

    FATAL - 0

    WARNING - 0

    SEVERE - 0

    Customer Hosts

    Notices

    NPP Maneuver at 11/27 14:55 EST. No data 14:45 to 15:30 EST

    1/4

    Log/Alert Summaries

    nde resource control panel
    NDE Resource Control Panel

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Resource Control

    STOP Server

    STOP Loop

    STOP Server

    STOP Loop

    STOP Server

    STOP Server

    STOP Loop

    STOP Loop

    STOP Loop

    START Server

    START Loop

    START Loop

    START Loop

    nde resource control panel1
    NDE Resource Control Panel

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Resource Control

    STOP Server

    STOP Loop

    STOP Server

    STOP Loop

    STOP Server

    STOP Loop

    STOP Server

    START Loop

    STOP Server

    STOP Loop

    START Server

    START Loop

    STOP Server

    STOP Loop

    START Server

    START Loop

    nde resource control panel2
    NDE Resource Control Panel

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Resource Control

    STOP Loop

    START Server

    START Loop

    nde resource control panel3
    NDE Resource Control Panel

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Resource Control

    +

    Modify Job Boxes – n06apn2

    STOP Loop

    START Server

    START Loop

    Save

    Cancel

    nde resource control panel4
    NDE Resource Control Panel

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Resource Control

    STOP Server

    STOP Loop

    STOP Loop

    STOP Server

    STOP Loop

    START Loop

    nde resource control panel5
    NDE Resource Control Panel

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Resource Control

    Run Manually

    Run Manually

    operator is it working
    Operator – Is it working
    • Main Dashboard read primarily by HelpDesk
      • TAL has the option to use it but will often skip to other screens
    • Dashboard provides an at-a-glance health and status of the NDE System
    nde dashboard nominal
    NDE Dashboard - Nominal

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Product Generation

    Distribution

    Ingest

    Clean-up

    24m ago

    41.6/86.2 TBs

    10,449

    9,731

    9,297

    SAN

    48% used

    Stats Period

    1,394

    837

    most recent

    639

    Backlog

    Throughput

    Backlog

    Throughput

    Backlog

    Throughput

    1 hr

    FATAL - 0

    SEVERE - 0

    WARNING - 18

    Customer Hosts

    Notices

    NPP Maneuver at 11/27 14:55 EST. No data 14:45 to 15:30 EST

    1/4

    Log/Alert Summaries

    operators log
    Operators’ Log
    • Ops/Help Desk to record any incidents, shift report, etc
      • Exact rules per EPDS policy
    • Log entries include the time and user automatically
    • Entries can have tags to aid in searching later – select one or more tags in the system or create a new one
    • Entries stored in the database
    • Ops Log GUI provides for searching and sorting options
      • By tags
      • By time range
      • By user
      • By text found in the log
      • Any combination of the above
    operators log1
    Operators’ Log

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Operators’ Log

    +

    +

    +

    +

    Fatal Error

    NWS

    DIS Error

    Push Host

    Log message:

    Log it!

    Tags:

    operations is something wrong
    Operations – Is something wrong?
    • Example Scenario
      • The remaining scenarios are located in the backup slides
        • Ingest Failed Count is high
        • Old granule files in incoming input directory
        • Not getting ancillary data (DDS)
        • Ingest is slow
        • Ingest Resource down
    • Available Drill-Downs
    ingest scenarios background
    Ingest Scenarios - Background
    • HelpDesk is watching the Dashboard
      • Indicators show status of Ingest servers (2), the retransmit application, and any product data sources (ESPC/DDS)
      • Annotated status bars show Backlog and Throughput
      • Failed Count – Ingest requests receiving a checksum mismatch, corrupt data file, corrupt crc file, duplicate, or unknown product message
      • Old Count – Files older than 1 orbit still residing in the Landing Zone (crc/notification not received, crc corrupt, duplicate, or other ingest exception)
    • Drill-Downs
      • Backlog/Throughput – By Data Source, shows status and ‘last hour’ granule file counts by Ingest server
      • Failed/Old Granule Files –Time/Type of failure, source, checksum (if applicable)
      • Product Granule File Counts by Orbit – Critical product counts, latest displayed first
    ingest scenario 1 failed count
    Ingest Scenario #1 – Failed Count
    • HelpDesk determines count on Dashboard is rising – any non-zero count warrants investigation
    • Failed/Old Drill-Down reachedby clicking on Failed Count hyperlink
    • HelpDesk writes Trouble Ticket (TT), informs TAL
    • TAL investigates/takes corrective action
    is 1 nde dashboard
    IS#1 - NDE Dashboard

    NDE_OPS

    NDE Data Handling System

    Ingest Scenario #1

    Annie User|Help

    Product Generation

    Distribution

    Ingest

    Clean-up

    24m ago

    41.6/86.2 TBs

    9,324

    9,831

    7,731

    SAN

    48% used

    3,019

    2,742

    Stats Period

    2,503

    most recent

    Backlog

    Throughput

    Backlog

    Throughput

    Backlog

    Throughput

    1 hr

    FATAL - 2

    WARNING - 18

    SEVERE - 3

    Customer Hosts

    Notices

    NPP Maneuver at 11/27 14:55 EST. No data 14:45 to 15:30 EST

    1/4

    Log/Alert Summaries

    is 1 failed ingest drill down
    IS#1 - Failed Ingest Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Failed Granules

    of 4

    1

    is 1 tal actions
    IS#1 - TAL Actions
    • If Checksum Mismatch
      • If retransmit count > 0 then writes TT, Calls IDPS
      • If retransmit count = 0 then writes TT, Reorders file
    • If Unknown Product
      • Checks filename against registered file patterns for similarities
        • Unregistered – writes TT, assigns to DPA
        • Unmatched but Similar – writes TT, further investigation (Pop the Hood) warranted
    • If Duplicate
      • examines counts/times, determines whether to Pop the Hood
    • If Corrupt file/crc
      • examines counts/times, determines whether to Pop the Hood
    old granule drill down
    Old Granule Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Total Count: 257

    Old Granules in Incoming Input

    of 4

    1

    backlog throughput drill down
    Backlog & Throughput Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Backlog & Throughput

    granule file counts by orbit
    Granule File Counts By Orbit

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Granule File Counts By Orbit

    pgs scenarios background
    PGS Scenarios - Background
    • Product Generation Scenarios Drill Downs
      • Scenarios in Backup Slides
    • Help Desk is watching the Dashboard
      • Indicators show status of PGS factory servers (2), and PGS Processors (n)
      • Annotated status bars show Backlog and Throughput
      • Available Counts
        • Failures – Non-zero return code
        • Excessive CPU – Job exceeds CPU threshold
        • Old – Job remains in Queued state
        • Expired – Job Spec exceeds ‘wait for input’ time
    • Primary Drill-Downs
      • Product Rule Summary – Status/Job counts summarized by product type
      • Job/Job Spec – Specific details of Jobs and Job Specs
      • Product Coverage – Product file indicator by coverage time
    product generation summary
    Product Generation Summary

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Product Generation Status Summary

    Links to

    Job Box Configuration Drill Down

    Links to

    PGS Job/Job Spec Drill Down

    w/ filter on Production Rule and Status

    Links to

    Production Rule Definition

    Drill Down

    of 4

    1

    pgs job job spec drill down
    PGS Job/Job Spec Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    PGS Job Spec & Job Status

    of 4

    1

    pgs job job spec drill down1
    PGS Job/Job Spec Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    PGS Job Spec & Job Status

    of 4

    1

    pgs job job spec drill down2
    PGS Job/Job Spec Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    +

    Job Spec & Job Details

    Back

    PGS Job Spec & Job Status

    Job Process ID

    56778

    Update Priority

    Requeue Job

    Terminate

    Expire Job

    Done

    of 4

    1

    product coverage drill down
    Product Coverage Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Product Coverage

    of 96

    1

    operations is something wrong1
    Operations – Is something wrong?
    • Distribution
      • Distribution Resources are down
      • Customer not getting data
        • Push
        • Pull
      • Customer not being notified
      • Distribution throughput is low
    distribution summary
    Distribution Summary

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Summary

    Links to

    Distribution Request Drill Down

    w/ filter on selected status

    Links to

    Subscription Coverage Drill Down

    distribution request
    Distribution Request

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Request

    Links to

    Subscription Drill Down

    Links to

    Distribution Request Details Modal Window

    distribution request1
    Distribution Request

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Request

    Links to

    Notification Drill Down

    distribution request2
    Distribution Request

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    +

    Back

    Distribution Request

    Distribution Request Details

    DPR Status

    Priority

    Size

    DJ Status

    COMPLETE

    FAILED

    Cancel

    Save

    subscription coverage
    Subscription Coverage

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Links to

    Gap Checker Drill Down

    w/ filter on that Product

    Back

    NCO_OPS Subscription Coverage

    Create a missing job by clicking here

    Of 96

    1

    notification
    Notification

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Notification Jobs

    notification1
    Notification

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    +

    Back

    Notification Jobs

    Notification Job Details

    Notification Status

    FAILED

    Cancel

    Save

    operations is something wrong2
    Operations – Is something wrong?
    • Infrastructure
      • Cleanup
        • SAN Cleanup not working
        • Internal RAID Cleanup Processing Nodes not working
        • Database Cleanup not working
        • All cases require TAL to Pop the Hood
      • SAN Storage Alert
        • Pop the Hood
      • RAID Processing Node Alert
        • Pop the Hood
    operations quiesce stop system
    Operations – Quiesce/Stop System
    • To cleanly quiesce the system:
      • Suspend the IDPS subscription(s) to NDE
      • Allow Ingest to work off its current backlog
      • Shutdown Ingest (loop and JBoss Server) in the Resource Control Panel
      • Restart IDPS subscriptions to ensure no data loss from IDPS
      • As PGS and DIS work off their backlogs, shut them down as well
      • Perform any necessary maintenance
      • Bring the system back up per standard procedure
    nde system notices
    NDE System Notices

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    System Status

    Turn ON

    Turn OFF

    Internal Notices

    +

    +

    Customer Notices

    +

    Title

    Description

    Internal

    Customer

    +

    agenda4
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    management reports
    Management Reports
    • This presents a recommendation for an initial set of reports
    • Recommend Management Reporting IPT to finalize Reports by February 28, 2012.
    • Report Creation and Access
    • Data Sources
    • Reporting Requirements
      • Throughput
      • Performance
      • Latency
      • Availability
    report creation
    Report Creation
    • Developed and Delivered from NDE Development Environment as Software Patch
    • Initial report selection delivered with Build 4/5
    • Report design based on XML inputs
    • Reports will be generated and distributed automatically on set schedule
    • Running a given report on demand and/or changing the report schedule will be handled via Jaspersoft
    report access
    Report Access
    • Standard periodic reports will be rendered in PDF and/or Excel and automatically sent to appropriate stakeholders
    • Reports on demand available to read only to all internal NDE Users via NDE GUI
    data sources
    Data Sources
    • Dashboard and Drill Downs from NDE Operational Database
    • Reports from hourly summaries extracted to a separate data warehouse
    • Operational Database contains 39 days worth of transaction data to support resolution of issues identified in the reports
    • Data warehouse contains history of all summarized data.
    • Data can be summarized
      • Hourly, daily, weekly, monthly, quarterly, yearly, etc.
      • By product type, product subtype, product, and customer
      • Processing Host, etc.
    availability data
    Availability Data

    Number of Minutes

    agenda5
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    architecture
    Architecture
    • COTS Evaluations
      • XML Editors
      • GUI Frameworks
      • Reporting/Visualization Tools
      • Metrics Gathering and Reporting
    • Implementation
      • GUI Framework
      • Reporting
      • Metrics Gathering and Reporting
    cots evaluation xml editors
    COTS Evaluation - XML Editors
    • May use Solers Recommendation of XML Spy if needed
    candidates frameworks
    Candidates Frameworks
    • Apache Struts
    • Spring MVC
    • JBoss Seam
    • Google Web Toolkit
    • Sencha Ext GWT

    Eliminated due to high configuration overhead.

    Eliminated b/c required repositories not available offline.

    decision sencha ext gwt
    Decision: Sencha Ext GWT
    • Works with existing software platforms
      • Hiberate, JBoss AS
    • Interactive, customizable charting
      • Eliminates the need for a separate BI Tool for Dashboards
    • Extensive, customizable widgets
      • Out-of-the-box grids, filtering, data binding, etc.
    • Provides consistency across ESPDS
      • Code re-use, knowledge transfer,lessons learned between NDE and PDA
    candidate reporting engines
    Candidate Reporting Engines
    • Crystal Reports
    • JasperSoft Enterprise
    • Actuate BIRT
    • Jasper Reports
    • Business Intelligence Reporting Tool (BIRT)

    Eliminated due to high cost ($20k and up) with limited added value for NDE (Sencha provides for the Dashboard)

    decision jasperreports
    Decision: JasperReports
    • Meets all requirements at the right price (free)
    • Works with existing software platforms
      • Hiberate, Superior integration with java
    • Larger Community and Userbase
      • Increased chance of finding quick solutions to roadblocks
      • Reduced chance of finding roadblocks due to product maturity
    • Superior Analytical Tools in the Open Source Product
      • Jaspersoft OLAP and ETL provide room to grow in analytics
    • Most cost effective path to a commercial support license if required in the future
    visualization tools
    Visualization Tools
    • Google Earth and NASA World Wind (Open Source)
    • Evaluation
      • World Wind better at handling large polygon shapes crossing the International Dateline
      • Google Earth Server behind firewall requires a commercial license
    • Decision: NASA World Wind
    metrics cots evaluation
    Metrics – COTS Evaluation
    • Orion selected over Hyperic after further evaluation
    • JBoss Operations Network selected for application level monitoring of JBoss Servers
    • Internal OS utilities such as sar, nmon, top, and timex utilized for real-time performance tuning and evaluation
    portal architecture
    Portal Architecture

    JBoss Application Server

    Presentation Layer

    UI Components (GWT Widgets)

    UI Process Components

    User

    Browser

    Business Layer

    Business Domain Components (DTO, Models)

    Business Logic Components (EJB)

    Data Layer

    Oracle DB

    Data Access Component (Hibernate)

    build 4 5 architecture
    Build 4/5 Architecture

    NDE/Production Zone

    Monitor and Control Services

    Web/App Server

    n25rpn07

    Web/App Server

    n25rpn08

    http

    • Sencha GUI Libraries
    • Jasper Reports Server
    • Jasper ETL
    • NASA World Wind
    • NDE Monitors
    • JON Server

    NDE Internal Users: Helpdesk Operator, TAL, PAL

    Oracle DB

    DMZ

    Customer Services

    Web/App Server

    n26rpn07

    Web/App Server

    n26rpn08

    https

    • Sencha GUI Libraries

    Customer

    monitoring framework
    Monitoring Framework

    NDE Host n

    Oracle Host

    NDE Info

    Services

    NDE and NOAA Management

    JVM

    Oracle Server

    JBoss

    NDE Data Warehouse (Historical)

    Reports

    • NDE Administrator
    • System
    • Engineering Staff
    • PAL Staff
    • Customers

    ETL

    NDE Service

    Dashboard

    NDE Class 1

    NDE Database

    (Operational)

    NDE Class n

    Oracle Enterprise Manager

    NDE Apps

    NDE Logs

    • Admins:
    • Database
    • System
    • Network
    • JBoss

    JBoss Operational Network (JON) Server

    JON Agent

    Orion Agent

    Orion Server

    metrics
    Metrics
    • JBoss Operations Network (JON)
      • Monitors resources at application level
      • Memory, CPU, Application Storage
      • Alerts appear on application dashboard
    • Orion / Oracle EM/ Cisco SAN
      • Infrastructure monitoring for Admin perspective
      • Alert to Admin
    • NDE Algorithm Execution
      • Utilized AIX timex command
      • Measures CPU, Memory, I/O per execution
    agenda6
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    security
    Security
    • System Security
    • Security Testing
    • Customer Access Control
    system security and testing
    System Security and Testing
    • NDE Resides within OSPO/ESPC networks and conforms to EPSC security procedures
      • All external connections pass through ESPC firewalls to the NDE DMZ and must pass ESPC firewall rules
      • All servers comply with ESPC security policies
      • All User IDs require ESPC approval
      • Customer Portal will be designed for security, with tightly regulated database access and no access to most NDE systems
    • Security Testing
      • NDE portals will undergo security testing (as also seen in the Testing section)
        • Scans will be made to find and eliminate SQL injection and other vulnerabilities
    customer access control
    Customer Access Control
    • OS Layer User IDs:
      • “Pull” customers have ftps/ftp-only IDs
        • These IDs are granted and controlled through standard ESPC procedures
        • These IDs only work for ftp(s) on pull servers in the DMZ
        • Again, these connections must pass ESPC firewall rules, including source IP
      • “Push” customers have no OS-level IDs
    • Data Access Control:
      • Subscriptions control what products the customer receives
      • Subscriptions set External Host Definitions for “Push” Customers
      • The Customer Profile in NDE defines points of contact for any necessary direct verifications and the trust level for Data Denial
    • Application Layer User IDs:
      • Customers have login access to a DMZ-hosted site (Customer Portal)
        • This site is read only – excepting their ability to change their own password
        • Site displays only data relevant to their subscriptions (includes general notices)
        • Site is encrypted
    internal user access control
    Internal User Access Control
    • OS Layer User IDs
      • Sys Admins and Engineers (TALs, Developers) will have direct login to many or all NDE Machines as befits their duties, controlled by standard ESPC procedures
      • PALs will have limited read access to the SAN and the processing node forensics directories and have the ability to VPN in to a server running Firefox to have remote access to the NDE internal site
    • Data Access Control
      • Internal Permissions are set to match role responsibilities
    • Application Layer User IDs (Internal Portal)
      • Internal Portal User IDs are granted via standard ESPC procedures
        • They are in addition to any OS-level IDs granted
        • The internal site can only be accessed from within the ESPC network
    nde internal portal data access
    NDE Internal Portal Data Access
    • NDE application layer access controlled by NDE:
      • Each User in the database is associated with a role or roles
      • The permissions associated with a role has are centralized in one java class, simplifying changes, if needed
      • Upon login, their session stores all the accesses provided by the user’s roles
        • NDE Security Agent (java class) provides the user permissions
        • Access checks are very fast as the user moves through the portal
      • Accesses will be defined at the page, widget, or even data item level as needed at design time
        • There are a limited number of updatable items, making the number of role controlled items reasonably small
      • Changing a user’s role is handled via an NDE User Management GUI, which handles the database updates
    access control nde security agent
    Access Control – NDE Security Agent

    NDE_OPS

    NDE_ADMIN

    NDE Data Handling System

    NDE Data Handling System

    • The user’s login session stores a list of all accesses
      • One page design serves multiple roles based on a single access check at login time

    (At Login Time)

    (At Page Access Time)

    User Session

    (Editable)

    UserID

    Admin User

    Access controlled item list

    ItemAllowed?

    UpdateUser No

    ViewMngUser No

    UpdateJobBox Yes

    ViewDashbrd Yes

    NDE Security Agent

    Ops User

    OR

    (View Only)

    DB

    manage users
    Manage Users

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Manage Users

    Reset

    Save

    agenda7
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    builds 4 5 integration test
    Builds 4&5 Integration Test
    • Develop an Integration Plan -Schedule builds of design items (aggregations of units associated with one or more CRs)
    • Develop thread diagrams of Customer Services, Monitoring and Control , etc.
    • Develop Detailed Integration Test Procedures, and review
    • Execute Integration Test Procedures
    • Develop success criteria for requirements verification
    builds 4 5 verification test
    Builds 4&5 Verification Test
    • Regression Test Build 1-3 capabilities – from Builds 1,2, and 3 SIT Procedures
    • Develop Requirements Verification Test Plan
    • Develop Requirements Verification Test Procedures from Integration Procedures
    • Present Test Readiness Review
    • Execute Requirements Verification Test Procedures
    test tools
    Test Tools
    • Web Automation Test Tools
      • Benefits -
        • Speed Regression testing – batch testing
        • Test compatibility/portability
      • Drawbacks
        • Testing human factors
        • GUI widget label updates can break recorded scripts
      • Examples
        • Selenium
        • HTT TEST
    agenda8
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    build 4 5 schedule
    Build 4/5 Schedule
    • Schedule Start End
      • Development Schedule 12/11 09/12
        • Develop detailed requirements
        • IPT Engagements (Complete NLT 03/31)
        • Prototyping
        • Coding
        • Unit/Integration Testing
      • Test Schedule 06/11 12/12
        • Establish infrastructure
        • Develop Test Plan/Procedures
        • Integration Testing
        • Regression Testing
        • Verification Testing
      • Documentation Schedule 12/11 03/13
    agenda9
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    agenda10
    Agenda
    • Background
    • Customer Services
    • Product Management
    • Operations
    • Management Reporting
    • Architecture
    • Security
    • Testing
    • Schedule
    • Stakeholder Requirements Met
    • Government Approval to Proceed
    team solers contact information
    Team Solers Contact Information
    • Address

    7474 Greenway Center Drive

    Suite 400

    Greenbelt, MD 20770

    • Phone
      • 240-473-0080 x100
    • Fax
      • 240-473-0080 x161
    • Email
      • daniel.beall@solers.com
      • jamison.hawkins@lmco.com
      • rsikorski@innovim.com
    xml file development
    XML File Development
    • Created in the Development Environment
    • Validated on SADIE

    Development Environment

    XML

    Authoring Tool

    XML File

    Database

    Validation

    nde patch contents for ingest
    NDE Patch Contents for Ingest

    Development

    Test

    OPS

    CCB

    Approval

    NDE Patch

    XML

    Authoring Tool

    NDE Patch

    XML File

    Validation

    Install

    Install

    DB

    DB

    DB

    • Product Description tables
    • External Product Data Source tables
    product definition ingest
    Product Definition - Ingest
    • Product Definition XML files include
      • Identifying file pattern
      • Data Provider
      • Short and long descriptions
      • Metadata extraction function/additional validation steps
      • Allows optional detail for automated tailoring services
      • SAN storage location
      • Test or Operational status
      • Allows categorization by product area, type
    product definition xml
    Product Definition XML

    <ProductDefinition

    ingestDirectoryName="incoming_input"

    productShortName=“VIIRS-M5TH-EDR“

    productLongName=“VIIRS 5th M Band Imagery EDR"

    productDescription=“The VIIRS Moderate Band Imagery EDRs are characterized by a 750m Horizontal Reporting Interval (HRI). All M-Band imagery products are re-sampled from the VIIRS moderate resolution SDR geolocation to a GTM projection."

    productMetadataLink=“/opt/apps/nde/pds/data/VIIRS-M5TH-EDR.xml“

    productProfileLink="“

    productType=“XDR"

    productIDPSMnemonic=“EDRE-VMOD-C0030“ productAvailabilityDate="“

    productCIPPriority ="“ productStatus=""

    productFilenamePattern=“VM05O_npp_d________t________e_______b_____c_________noaa____.h5“

    productFilenamePrefix=“VM05O"

    productHomeDirectory="products/VIIRS-M5TH-EDR“

    productSpatialArea="“ productIDPSRetransmitLimit=“5“ productArea=""

    </ProductDefinition >

    product generation
    Product Generation
    • Receive Delivered Algorithm Package (DAP)
    • Integrate in SADIE
    • Promote to ALGTEST/Functional testing
    • Promote to POP/Performance testing
    • Approve for promotion to OPS
    • Install into OPS via delivered software patch including associated XML files
    nup tp via dap transition to ops1
    NUP/TP via DAP Transition to Ops

    STAR

    Integration (SADIE, IPT)

    Functional Test (ALGTEST, Test)

    Performance Test (POP, Test)

    CCB (Approve Promotion to OPS)

    XML Files

    Install into OPS

    NDE Patch

    DAP

    • Package contains all SPSRB documentation (build instructions, release notes)
    • Patch is created from ‘built’ software
    nde tp transition to operations
    NDE-TP Transition to Operations

    Integration (SADIE, IPT)

    Functional Test (ALGTEST, Test)

    Performance Test (POP, Test)

    CCB (Approve Promotion to OPS)

    Install into OPS

    NDE Patch

    • Encapsulated within Production Rule (DSS)
    • Script (IDL)
    • or Both
    nde patch contents for product sys1
    NDE Patch Contents for Product Sys

    DAP

    Development

    Test

    OPS

    CCB

    Approval

    NDE Patch

    XML

    Authoring Tool

    NDE Patch

    XML File

    Validation

    Install

    Install

    DB

    DB

    DB

    PGS Nodes

    PGS Nodes

    • Product Description tables
    • Algorithm/Production Rule tables
    • External Product Data Source tables
    nde patch contents for product sys2
    NDE Patch Contents for Product Sys

    Database

    NDE Patch

    XML files

    • Product Description tables
    • Algorithm/Production Rule tables
    • External Product Data Source tables

    INSTALL

    Algorithms

    Processing Nodes

    nde xml configuration files
    NDE XML Configuration Files
    • Product Definitions
    • Product Profile Definitions
    • Enterprise Measures Definitions
    • Algorithm/Rule Definitions
    • Priorities PGS/DIS
    • Job Size PGS/DIS
    • Gazetteer XML (Spatial AOI)
    • NDE Support Functions
    • Product Group
    • Ingest Incoming Directory
    • Platform
    • Platform Sensor
    • Deliver Notification Type
    • Data Denial Flag
    production rule xml
    Production Rule XML

    <ProductionRule

    prRuleName="NUCAPS Retrieval Granule" prRuleType="Granule"

    prStartBoundaryTime="null" prProductCoverageInterval_DS="null"

    prRunInterval_DS="null“ prTemporarySpaceMB="1"

    prEstimatedRAM_MB="1" prEstimatedCPU="120"

    prProcessingStart="60" prCoverage="60"

    prActiveFlag="1"

    jobPriority="Medium"

    jobSize="Small"

    prWaitForInputInterval="interval '101' minute"

    gzFeatureName="null">

    <Algorithm algorithmName="NUCAPS_Retrieval.pl" algorithmVersion="1.0"/>

    <ProductionRuleParameters>

    <PRParameteralgoParameterName="OPS_BIN” prParameterValue=“~/NUCAPS/OPS/Common_Bin"/>

    </ProductionRule>

    algorithm xml
    Algorithm XML

    <Algorithm

    algorithmName="NUCAPS_Retrieval.pl" algorithmVersion="1.0"

    algorithmCommandPrefix="/usr/bin/perl -w“

    algorithmExecutableLocation="NUCAPS/OPS/scripts"

    algorithmNotifyOpSeconds="60"

    algorithmType="Science"

    algorithmPcf_Filename="NUCAPS_Retrieval.pl.PCF"

    algorithmPsf_Filename="NUCAPS_Retrieval.pl.PSF"

    algorithmLogFilename="NUCAPS_Retrieval.pl.log"

    algorithmLogMessageContext="."

    algorithmLogMessageWarn="WARNING:"

    algorithmLogMessageError="Error in" >

    <AlgorithmParameters>

    <AlgoParameteralgoParameterName="OPS_BIN" algoParameterDataType="string"/>

    </Algorithm>

    enterprise measures xml
    Enterprise Measures XML

    <ndeEnterpriseSchema>

    <EnterpriseDimensions>

    <EnterpriseDimension name=“AlongTrack-1536-1” start=“0” interval=“1” end=“*” storageSize=“1536” storageMaxSize=“1536” />

    <EnterpriseDimension name=“AlongTrack-1541-1” start=“0” interval=“1” end=“*” storageSize=“1541” storageMaxSize=“1541” />

    <EnterpriseDimension name=“BeamPosition-96-0” start=“0” interval=“1” end=“96” storageSize=“96” storageMaxSize=“96” />

    <EnterpriseDimension name=“Bulk SST Scale/Offset-2-1” start=“0” interval=“1” end=“*” storageSize=“2” storageMaxSize=“2” />

    <EnterpriseDimension name=“Channel-22-0” start=“0” interval=“1” end=“22” storageSize=“22” storageMaxSize=“22” />

    </ndeEnterpriseSchema>

    product profile xml
    Product Profile XML

    <NPOESSDataProduct>

    <ProductName>ATMS Science SDR: Channels 1 through 22</ProductName>

    <CollectionShortName>ATMS-SDR</CollectionShortName>

    <DataProductID>SATMS</DataProductID>

    <ProductData>

    <DataName>ATMS SDR Product Profile</DataName>

    <Field>

    <Name>BeamTime</Name>

    <Dimension>

    <Name>Scan</Name>

    <GranuleBoundary>1</GranuleBoundary>

    <Dynamic>0</Dynamic>

    <MinIndex>12</MinIndex>

    <MaxIndex>12</MaxIndex>

    </NPOESSDataProduct>

    gazetteer xml
    Gazetteer XML

    <Gazetteer

    gzFeatureName=“AWIPS ATLANTIC”

    gzLocationSpatial=“SDO_GEOMETRY(2003, NULL, NULL, SDO_ELEM_INFO_ARRAY(1, 1003, 1), SDO_ORDINATE_ARRAY(-120.000, 75.000, 5.000, 75.000, 10.000, 75.000, 10.000, 10.000, 10.000, -50.000, 5.000, -50,000, 5.000, -50.000, -120.000, -50.000, -120.000, 10.000, -120.000, 75.000))”

    gzSourceType=“”

    gzLocationElevationMeters=“0”

    gzDesignation=“AOI”

    </Gazetter>

    product quality summary report
    Product Quality Summary Report
    • “Detect and report to operators and management the acceptability of [input and output product] metadata according to configurable thresholds…”
    • Product Quality Summary Capture - Detect
      • Ingest
        • xDR metadata names/values extracted and cataloged
        • Framework to capture ancillary metadata if necessary
      • Product Generation
        • Production Rules incorporate quality threshold conditionals
      • Distribution
        • Subscriptions incorporate quality threshold conditionals
    • Product Quality Summary - Report
      • Online and printed by product summary of file counts and percentages for value ranges
    ingest scenario 2 old granules
    Ingest Scenario #2 – “Old” granules
    • HelpDesk determines Old count is rising, an old granule file is > 1 orbit (102 minutes) old
      • HelpDesk ignores only if NDE is working off a known backlog
    • Failed/Old Drill-Down reachedby clicking on Old Count hyperlink
    • HelpDesk writes Trouble Ticket (TT), informs TAL
    • TAL investigates/takes corrective action
    is 2 old granule drill down
    IS#2 - Old Granule Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Old Granules in Incoming Input

    of 4

    1

    is 2 tal actions
    IS#2 - TAL Actions
    • If “No crc file received” – potential problem with the data provider
      • Determines product validity
        • Valid: Writes TT, “fixes” by generating crc file
        • Invalid: Writes TT/Contacts data source/removes file
    • If “crc file processed” – potential DHS problem
      • Renames/Moves existing crc file
      • Determines extent of the problem
      • Writes TT, determines whether to Pop the Hood
    ingest scenario 3 not getting data
    Ingest Scenario #3 – Not Getting Data
    • HelpDesk sees Dashboard indicating Backlog low and Throughput down, products are not being created
      • Ignores only if during a known anomaly – ESPC/DDS down (indicator is red)
    • Backlog & Throughput Drill-Down reached by clicking on either Backlog or Throughput status bars
    • Product Granule File Counts By Orbit reached by clicking Critical Files Late Count or B & T Drill-Down link
    • HelpDesk investigates, writes TT, informs TAL
    • TAL investigates/takes corrective action
    is 3 backlog throughput drill down
    IS#3 - Backlog & Throughput Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Backlog & Throughput

    is 3 backlog throughput drill down1
    IS#3 - Backlog & Throughput Drill Down

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Backlog & Throughput

    is 3 granule file counts by orbit
    IS#3 - Granule File Counts By Orbit

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Granule File Counts By Orbit

    is 3 tal actions
    IS#3 - TAL Actions
    • Using Drill-Downs TAL determines which product is the missing, data source status also shown
    • If DDS
      • Up – Writes TT, contacts DDS TAL to determine why ancillary products are unavailable
      • Down – Writes TT, contacts DDS TAL for further info
    • If IDPS – Writes TT, calls IDPS
    • If NDE-PGS – DHS ‘internal’ cause, Check PGS, Pop the Hood/take corrective action
    ingest scenario 4 ingest is slow
    Ingest Scenario #4 – Ingest is Slow
    • Help Desk determines Backlog is increasing /Throughput is decreasing or both, using backlog and throughput status bars
    • Help Desk ignores only during a known anomaly – maneuver, instrument downtime, other outage (communications)
    • Help Desk determines the count of Fatal messages in the Alert Summary is non zero (or has gone up)
    • Backlog & Throughput Drill-Down reached by clicking on status bar (see above)
    • Product Granule File Counts By Orbit reached by clicking on Critical Files Late Count or B & T Drill-Down link (see above)
    • Help Desk writes Trouble Ticket (TT), informs TAL
    • TAL investigates (determines scope)/takes corrective action
    is 4 tal actions
    IS#4 - TAL Actions
    • Scope determined using both Drill-Downs – which products/Data Sources
    • If no Ingest Resource Down – both servers green, all data sources available
      • Check Alert Summary log on Dashboard
      • If servers are ‘balanced’
        • Further investigation – JBoss Server logs, host resources, etc.
      • If servers are ‘unbalanced’, writes TT, call JBoss Admin
    ingest scenario 5 resource down
    Ingest Scenario #5 - Resource Down
    • HelpDesk determines which resource is causing the problem via Dashboard indicators (red) and/or alert messages
    • HelpDesk ignores only during a known anomaly – maneuver, instrument downtime, other outage (communications)
    • Depending upon which resource, HelpDesk writes TT and notifies TAL and/or SME for that resource
    pgs scenarios background1
    PGS Scenarios - Background
    • Help Desk is watching the Dashboard
      • Indicators show status of PGS factory servers (2), and PGS Processors (n)
      • Annotated status bars show Backlog and Throughput
      • Available Counts
        • Failures – Non-zero return code
        • Excessive CPU – Job exceeds CPU threshold
        • Old – Job remains in Queued state
        • Expired – Job Spec exceeds ‘wait for input’ time
    • Drill-Downs
      • Product Rule Summary – Status/Job counts summarized by product type
      • Job/Job Spec – Specific details of Jobs and Job Specs
      • Product Coverage – Product file indicator by coverage time
    pgs scenarios
    PGS Scenarios
    • (Placeholder to repeat the Dashboard slide)
    pgs scenario 1 nup tp not created
    PGS Scenario #1 – NUP/TP Not Created
    • Help Desk notices anomaly
      • Resource is not available
      • Backlog status bar is increasing (Throughput status bar may be decreasing)
        • Jobs Specs incomplete (waiting for an input file)
        • Jobs Queued (awaiting execution)
      • Any of the available counts are increasing
    • Help Desk writes TT, informs TAL
    • Production Rule Summary Drill-Down reached by clicking on Backlog/Throughput status bar(s)
    • TAL investigates/takes corrective action
    ps 1 production rule dd
    PS#1 - Production Rule DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Production Rule Status Summary

    of 4

    1

    ps 1 production rule dd1
    PS#1 - Production Rule DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Product Generation Status Summary

    Links to

    Job Box Configuration Drill Down

    Links to

    PGS Job/Job Spec Drill Down

    w/ filter on Production Rule and Status

    Links to

    Production Rule Definition

    Drill Down

    of 4

    1

    ps 1 pgs job job spec dd
    PS#1 - PGS Job/Job Spec DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    PGS Job Spec & Job Status

    of 4

    1

    ps 1 tal actions
    PS#1 - TAL Actions
    • If PGS Resource Down
      • Navigate to the Resource Control Panel Drill-Down and fix
    • Determine the extent of the problem – single product/all products
    • If single product see Scenario #3
    • If PGS operating slowly see Scenario #2
    • If all products Check Ingest
    pgs scenario 2 pgs is too slow
    PGS Scenario #2 – PGS is too Slow
    • Help Desk determines Backlog increasing/Throughput decreasing
    • Help Desk writes TT, informs TAL
    • TAL investigates/takes corrective action
      • To determine which products are not being made, use Check Chart drill down
      • Production Rule Summary Drill-Down reached by clicking on Backlog/Throughput status bar(s)
    ps 2 product coverage dd
    PS#2 - Product Coverage DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Product Coverage

    Of 96

    1

    ps 2 production rule dd
    PS#2 - Production Rule DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Production Rule Status Summary

    of 4

    1

    ps 2 pgs job job spec dd
    PS#2 – PGS Job/Job Spec DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    PGS Job Spec & Job Status

    of 4

    1

    ps 2 pgs job job spec details
    PS#2 - PGS Job/Job Spec Details

    NDE_OPS

    NDE Data Handling System

    Annie User | Help

    PGS

    Ingest

    Distribution

    Other

    Dashboard

    Gap Checker

    Update Priority

    Requeue Job

    Back

    Terminate

    Expire Job

    PGS Job Spec & Job Status

    Job Spec & Job Details

    X

    40

    Alg Process ID: 56778

    of 4

    1

    ps 2 tal actions
    PS#2 - TAL Actions
    • Corrects Jobs using excessive CPU time (slowing everything else down) – Terminates particular jobs using Job Spec/Jobs Details page
    • Retries terminated jobs and monitors, updates TT
    • Determines if enough job boxes are available – Update Job Box Configuration Drill-Down
    pgs scenario 3 some prod not created
    PGS Scenario #3 - Some Prod not created
    • Help Desk sees Expired or Old counts increasing, contact from customer
    • Help Desk writes TT, informs TAL
    • TAL investigates/takes corrective action
      • Check Chart reached by Menu
      • Job/Job Spec Drill-Down reached by clicking on Expired/Old counts
      • Production Rule Summary Drill-Down reached by clicking on Backlog/Throughput status bar(s)
      • Production Rule Details reached by clicking on Production Rule Name
    ps 3 pgs product coverage dd
    PS#3 - PGS Product Coverage DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Product Coverage

    Of 96

    1

    ps 3 pgs job job spec dd
    PS#3 - PGS Job/Job Spec DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    PGS Job Spec & Job Status

    of 4

    1

    ps 3 production rule dd
    PS#3 - Production Rule DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Production Rule Status Summary

    of 4

    1

    ps 3 tal actions
    PS#3 - TAL Actions
    • Determines if Job Specs are expiring or Job Specs are waiting for input
      • Clicks Expired count to determine rule
      • Clicks Product Coverage Drill-Down to determine which input product is missing (reason for expiration)
      • Check ingest for input product arrival
      • Update TT
    • Determines if Production Rule is active
      • Clicks Production Rule Name to display Production Rule Details
      • If Rule is inactive, updates, updates TT
    pgs scenario 4 jobs failing not picked up
    PGS Scenario #4 – Jobs Failing/Not Picked up
    • Help Desk sees Failed or Old counts increasing or the Backlog status bar increasing
    • Help Desk writes TT, informs TAL
    • TAL investigates/takes corrective action
      • Job/Job Spec Drill-Down reached by clicking on Failed/Old counts
      • Production Rule Summary Drill-Down reached by clicking on Backlog/Throughput status bar(s)
      • Check Log/Alerts Summary for processing node device utilization
    ps 4 pgs job job spec dd
    PS#4 - PGS Job/Job Spec DD

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    PGS Job Spec & Job Status

    of 4

    1

    ps 4 pgs job job spec details
    PS#4 - PGS Job/Job Spec Details

    NDE_OPS

    NDE Data Handling System

    Annie User | Help

    PGS

    Ingest

    Distribution

    Other

    Dashboard

    Gap Checker

    Update Priority

    Requeue Job

    Back

    Terminate

    Expire Job

    PGS Job Spec & Job Status

    Job Spec & Job Details

    X

    40

    Alg Process ID: 56778

    of 4

    1

    ps 4 tal actions
    PS#4 - TAL Actions
    • Determines type/extent of Failures from Production Rule Summary Drill-Down
      • Determines Job error code details by clicking on specific Job Id
      • Pops the Hood (examine forensics, logs, etc.), updates TT
    • If processing node device Alerts show out of space
      • Clean up the forensics directories of old data to allow processing, updates TT
    • Determines why Jobs are getting Picked Up – stay in QUEUED state
      • Determines if enough job boxes are available – Update Job Box Configuration Drill-Down
    operations is something wrong3
    Operations – Is something wrong?
    • Distribution
      • Distribution Resources are down
      • Customer not getting data
        • Push
        • Pull
      • Customer not being notified
      • Distribution throughput is low
    distribution scenarios
    Distribution Scenarios
    • (Placeholder to repeat the dashboard slide for visual reference)
    dis scenario 1 resources down
    DIS Scenario #1 - Resources down
    • Operator notices one or more DIS resources to be red on Resource status indicators, or a resource related alert has been triggered, writes TT informs TAL
      • Throughput may not be affected
      • TAL resolves a red indicator with the following, per the indicator:
        • Factory - Restart
        • Processor Node - Restart
        • ftps Pull Server – Notify Sys Admin, update TT
        • Customer push server – Contact to resolve connection issue, if needed involve Network Administrator
      • To resolve a resource status alert the TAL will
        • Pop the hood
    dis scenario 2 cust not getting data
    DIS Scenario #2 – Cust not getting data
    • Customer calls and informs operator/helpdesk that data is not being received, Operator/TAL initiates one or more of the following to resolve the problem
    • Operator checks status of Distribution Resources
      • See Scenario #1
    • TAL checks if Distribution Jobs/Requests failing and/or not created
      • TAL drills down to Subscription Coverage and examines
        • If Jobs/Requests fail, TAL investigates and attempts to resolve
        • If Jobs/Requests absent, checks Ingest/PGS Coverage; if coverage is OK, investigate
    • TAL Checks Subscription’s active flag and check Subscription’s job boxes
      • TAL drills down to Subscription Details and updates the Active flag (if deactivated)
      • TAL checks the Job Boxes on the Resource Control Panel
    ds 2 distribution summary
    DS#2 - Distribution Summary

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Summary

    ds 2 distribution summary1
    DS#2 - Distribution Summary

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Summary

    Links to

    Subscription Coverage Drill Down

    Links to

    Distribution Request Drill Down

    w/ filter on selected status

    ds 2 distribution request
    DS#2 - Distribution Request

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Request

    ds 2 distribution request1
    DS#2 - Distribution Request

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Request

    Links to

    Subscription Drill Down

    Links to

    Distribution Request Details Modal Window

    ds 2 distribution request2
    DS#2 - Distribution Request

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Request

    ds 2 distribution request3
    DS#2 - Distribution Request

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Request

    Links to

    Notification Drill Down

    ds 2 distribution request4
    DS#2 - Distribution Request

    NDE_OPS

    NDE Data Handling System

    Annie User | Help

    PGS

    Ingest

    Distribution

    Other

    Dashboard

    Gap Checker

    Distribution Request Details

    X

    Cancel

    Save

    Back

    Distribution Request

    COMPLETE

    40

    40

    FAILED

    ds 2 subscription coverage
    DS#2 - Subscription Coverage

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    NCO_OPS Subscription Coverage

    Of 96

    1

    ds 2 subscription coverage1
    DS#2 - Subscription Coverage

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    NCO_OPS Subscription Coverage

    Links to

    Gap Checker Drill Down

    w/ filter on that Product

    Of 96

    1

    ds 2 customer data guis
    DS#2 - Customer Data GUIs

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Subscription Definitions

    Reset

    Save

    dis scenario 3 cust not notified
    DIS Scenario #3 – Cust not notified
    • Customer calls and informs operator/Help Desk that the data delivery notifications are not being received, TAL performs one or more of the following to resolve the problem
    • Operator checks Notification Processors status from Distribution Resource indicator or sees Customer Notification Server indicator is red
      • See Scenario #1
    • Operator checks whether Notification is specified for the Subscription, TAL sets the Notification mechanism for the Subscription, if not already, to resolve the problem
    • TAL checks whether the products are being created/delivered in the first place (refer to Scenario #2)
    • TAL drills down to Distribution Job details of that customer and checks the Notification Job Status
      • TAL investigates any Notification Job failures and attempts to resolve the problem
    ds 3 distribution request
    DS#3 - Distribution Request

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Distribution Request

    Links to

    Notification Drill Down

    ds 3 notification
    DS#3 - Notification

    NDE_OPS

    NDE Data Handling System

    Annie User | Help | Logout

    Back

    Notification Jobs

    ds 3 notification1
    DS#3 - Notification

    NDE_OPS

    NDE Data Handling System

    Annie User | Help

    PGS

    Ingest

    Distribution

    Other

    Dashboard

    Gap Checker

    Cancel

    Save

    Back

    Notification Job

    Notification Job Details

    X

    FAILED

    dis scenario 4 low throughput
    DIS Scenario #4 - Low throughput
    • Operator notices low DIS throughput from Status bars, informs TAL, TAL performs one or more of the following to resolve the problem
    • TAL examines if Ingest/PGS throughput is low (See appropriate scenarios in Ingest/PGS)
    • TAL drills down to Subscriptions and ensures their active flag is set accordingly (Jobs are never created for inactive Subscriptions)
    • TAL drills down to Distribution Job Details and:
      • examines Distribution request details to see if Distribution factory is slow in completing Distribution Requests/Jobs
        • TAL investigates and attempts to resolve the problem
        • TAL informs Sys/Network Admin if investigation suggests a network issue
    • TAL checks Distribution processors status from Resource Status indicators Alerts on the Log/Alerts Summary portion of the Dashboard
      • if they are down, investigates and attempts to bring them up (same as in Scenario #1)