Metrics and monitoring capabilities for earth science data systems
Download
1 / 28

Metrics and Monitoring Capabilities for Earth Science Data Systems - PowerPoint PPT Presentation


  • 100 Views
  • Uploaded on

Metrics and Monitoring Capabilities for Earth Science Data Systems. ESDSWG Wilmington, Delaware October 20-22, 2009. Outline. Core and Community Capabilities Network Flow Requirements and Monitoring Science Data Production and Distribution Transitioning from Community to Core.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Metrics and Monitoring Capabilities for Earth Science Data Systems' - chet


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Metrics and monitoring capabilities for earth science data systems

Metrics and Monitoring Capabilities for Earth Science Data Systems

ESDSWG

Wilmington, Delaware

October 20-22, 2009


Outline
Outline Systems

  • Core and Community Capabilities

  • Network Flow Requirements and Monitoring

  • Science Data Production and Distribution

  • Transitioning from Community to Core


Core and community
Core and Community Systems

  • Core

    • Data system elements needed to ensure processing, archival and distribution of data collected by EOS designated Earth science missions in a timely and usable manner

  • Community

    • Data system elements developed and deployed largely outside the NASA core elements characterized by ‘evolvability’ and innovation with the potential to be integrated into the core


Metric and monitoring capability comparison
Metric and Monitoring Capability Comparison Systems

* Limited Capabilities. Sometimes core/community overlap.


Network flow requirements
Network Flow Requirements Systems

  • Science data flow requirements are formulated to support nominal and reprocessing efforts.

  • Science teams, processing and archival facilities, among others’, are responsible for requirement development.

  • Requirements are stored in documents under configuration control.

  • Requirements are constantly evaluated.


Network monitoring
Network Monitoring Systems

  • Includes both passive and active network monitoring tools that gather network statistics, populate the information into a database, and provides tools to analyze the data to:

    • Assist in troubleshooting performance problems

    • Track utilization of network resources

    • Verify requirements against actual

    • Help forecast required upgrades


Passive Monitoring System Overview Systems

Data

Center

Limited access

EBnet Engineering hosts

Data

Center

EBNet

Data

Center

ENSIGHT

DB Collector

Data

Center

Secured ENSIGHT

Web Server

Proxy Web Server

Passive performance data (utilization, CPU pct., NetFlow, etc.) provided to collector

Passive monitoring information (graphics, statistics, HTML) pushed to secured web server

Proxied Performance web-site (HTML)

Interactive Live-monitoring, Flow graphs and reports


SNMP Object Monitoring Systems

  • Similar to MRTG

  • Data stored and available via SQL

  • Web front-end permits control of collection and graphing


Custom Flow Graph Systems

  • Examination of FTP transfers between two networks

  • One hour time period examined

  • Ex.: Used to troubleshoot slow FTPs, and exonerate network


Tracking NetFlow Impact on Network Resources Systems

  • Network Flow load is tracked on local LAN

  • TCPdump data is collected, extrapolated

  • Graph indicates load on WAN, LAN caused by Network Flow

  • Rarely more than 15K bits/ second for 4 routers


Ensight active testing overview
ENSIGHT Active Testing Overview Systems

  • End-to-end user level test

    • Little or no visibility into network internals

  • Purposes

    • Assess whether networks as implemented meet EOS requirements

    • Assess whether existing networks can support intended applications

    • Resolve user complaints:

      • Network problems -- or somewhere else??

      • Determine bottlenecks -- seek routing alternatives

    • Provide a basis for allocation of additional resources


Active testing system overview
Active Testing: System Overview Systems

Test Source 1

Test Destination 1

o

o

o

o

o

o

Test Source 30

Test Destination 80

ENSIGHT

Users

Security Perimeter

Secured ENSIGHT Web Server, and

B/U database

ENSIGHT

Active Collectors

(Primary and B/U)

ENSIGHT

Database

(primary)

End-to-end iperf (or other) active network performance measurement

Active performance measurement results (throughput, packet loss, etc.) provided to collector

Periodic SCP fetch of external Active performance results

Performance graphics provided to secured web server

Performance web-site (HTML)


Integrated charts
Integrated Charts Systems

  • The problem: Neither iperf nor MRTG alone is sufficient to characterize the performance of a circuit

    • MRTG will be low if users are idle

    • But Iperf results will appear low if competing with active user flows

  • Solution: Add the iperf and MRTG measurements together.

    • But there are some difficulties

  • Improved Solution: Add the iperf and applicable Flow data

    • Flow data can be obtained for small time periods

    • But still susceptible to interference


Production and distribution monitoring
Production and Distribution Monitoring Systems

  • ESDIS Metric System (EMS) is used to track core components.

  • MCT is used to track community projects (e.g. MEaSUREs)


Ems overview
EMS Overview Systems

  • Automated collection, lookup, QA and reporting

  • Web based reporting interface

Users


Ems implementations
EMS Implementations Systems

OGPB TBD


Fy08 overview
FY08 Overview Systems



Fy08 science distribution
FY08 Science Distribution Systems

*OBPG data taken from the Ocean Color web site


Distribution trends
Distribution Trends Systems

*OBPG data taken from the Ocean Color web site




Mct overview
MCT Overview Systems

  • Web form for capturing metrics from community projects.

  • Projects manually enter data into form, typically monthly.

  • Web-based reporting interface for Program sponsors, PIs and NASA Managers.

  • Community recommends modifications to metric questions annually at the ESDSWG.


Mct transition
MCT Transition Systems

  • The Metric Collection Tool is being replaced.

  • Why Change

    • Consolidate data, support and sustaining engineering.

    • Ensure archival of metrics.

    • Single metric reporting interface.

  • When

    • Prototype now available.

    • Soliciting comments and testers.

    • Hope to have the tool available in January 2010.

  • Help us name it ….


Transitioning community to core
Transitioning Community to Core Systems

  • Some community projects datasets, tools and/or services may transition to core capabilities.

  • Technical approaches to handle this transition (from a metrics perspective) are being implemented through the consolidation of metric gathering tools.



ad