Sphinx Server - PowerPoint PPT Presentation

Slide1 l.jpg
Download
1 / 12

Current Sphinx Client/Server. Sphinx Server. Sphinx Client. Data Warehouse. Message Interface. Message Interface. Multi-threaded Control Process. DAG Reducer. Prediction Engine. Submitter . Planner. Generic Grid Site. Resource. Monitoring Service. Information Gatherer.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Sphinx Server

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Slide1 l.jpg

Current Sphinx Client/Server

Sphinx Server

Sphinx Client

Data Warehouse

Message Interface

Message Interface

Multi-threaded

Control Process

DAG Reducer

Prediction Engine

Submitter

Planner

Generic Grid Site

Resource

Monitoring Service

Information Gatherer


Sphinx client server with v2 components l.jpg

Sphinx Client/Server with V2 Components

Sphinx Server

Sphinx Client

Data Warehouse

Message Interface

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Submitter

Admission Control

Planner

Generic Grid Site

Resource

Data Management

Monitoring Service

Information Gatherer


Stand alone sphinx server with light client l.jpg

Stand alone Sphinx Server with light Client

Sphinx Server

Sphinx Client

Data Warehouse

Message Interface

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Generic Grid Site

Submitter

Resource

Data Management

Monitoring Service

Information Gatherer


Data warehouse l.jpg

Data Warehouse

Sphinx Server

  • Message Tables

  • Policy Information Tables

  • Entity Accounting Tables

  • Work (DAG, Job, etc) Tables

    • State

      • Unreduced

      • Unpredicted

      • Unaccepted (not admitted)

      • Unplanned (not yet feasible)

      • Unsent (assigned)

      • Unfinished (executing)

      • Remove (accounted or rejected)

    • Entity

    • Resource Property Requirements

    • Dependency Requirements

      • Do not start before “event”

      • I/O

    • QoS Requests

    • Resource Assignment for next k steps

  • Application Tables (in general, distributed…)

    • Transformation Catalogue

    • Profiles (CPU, Disk, Bandwidth requirements)

    • History (CPU, Disk, Bandwidth, actual use)

  • Data Tables (in general, distributed…)

    • Replica Catalogue (from RLS)

    • Profiles (Size, Bandwidth requirements)

    • History (Access rate, etc)

  • Resource Property Tables

    • CPU, Disk, Network Connection, middleware configuration, etc

  • Grid Weather Tables

    • CPU View

    • Storage View

    • Bandwidth View

Data Warehouse

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Submitter

Data Management

Information Gatherer

What about including Virtual Data Tables?


Message interface component l.jpg

Input (incoming)

DAG

Requirements

QoS requests

Status requests

Output (incoming)

DAG Table (“unreduced”)

Job Table (“unreduced”)

application

Requirements

QoS

Input (outgoing)

Message

Output (outgoing)

Status updates

Message Interface Component

Sphinx Server

Data Warehouse

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Submitter

Data Management

Information Gatherer


Dag reducer decomposer l.jpg

DAG Reducer (Decomposer?)

Sphinx Server

  • Input

    • DAG (“unreduced”)

    • Available (existing) data

  • Action

    • Remove DAG nodes for which input data exists

    • Add Jobs to Job table

  • Output

    • Pruned DAG (“unpredicted”)

    • Jobs (“unpredicted”)

Data Warehouse

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Submitter

Data Management

Information Gatherer


Prediction engine component l.jpg

Prediction Engine Component

Sphinx Server

  • Input:

    • “unpredicted” Job

    • Requirements (if available)

      • CPU hours

      • Storage hours

      • Bandwidth

    • Profile/History

  • Output:

    • Estimated Resource Usage

    • “unaccepted” job

Data Warehouse

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Submitter

Data Management

Information Gatherer


Admission control component l.jpg

Admission Control Component

Sphinx Server

  • Input

    • Entity Account Information

    • “unaccepted” DAG Request

      • Requirements

      • QoS

    • Predictions of Jobs (decomposed DAG) Resource Usages

    • Policy Constraints

    • Current schedule for next k steps

  • Output

    • Reject DAG Request

      • “Remove” DAG

      • “Remove” Jobs

    • Accept DAG with Estimated QoS

      • “Unplanned” DAG

      • “Unplanned” Jobs

Data Warehouse

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Submitter

Data Management

Information Gatherer


Planning component l.jpg

Planning Component

Sphinx Server

  • Job Planner (independent “work” that can be fully resolved onto resources at time t)

    • Input

      • “unplanned” Job Information

        • Requirements

        • Dependencies (I/O)

      • Job QoS Requests

      • Entity Account Information

      • Policy Constraints

      • Available Data

      • Available Applications

      • Available Resource Properties

      • Prediction of Job Resource Usage

    • Output

      • QoS Deliverable

      • “unsent” Job(s) schedule for the next k steps

  • DAG Planner (intra-dependent “work” that can not all be fully resolved onto resources at time t)

    • Input

      • “unplanned” DAG Information

        • Requirements

      • DAG QoS Requests

      • Current Job Schedule for next k steps

    • Output

      • QoS Deliverable

      • Modified Job Schedule for next k steps

Data Warehouse

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Submitter

Data Management

Information Gatherer


Job submission component also part of sphinx client l.jpg

Job Submission Component (also part of Sphinx Client)

Sphinx Server

  • Input

    • Fully planned Job:

      • Job state “unsent”

      • Assigned gatekeeper and jobmanager

      • I/O data movement locations

  • Action

    • Construct Job DAG

      • Set up environment

      • Import input data (if necessary)

      • Publish to DMC (if necessary)

      • Run application

      • Export output data (if necessary)

      • Publish to DMC (if necessary)

      • Cleanup

        • environment

        • Local I/O data (if necessary)

    • Submit to Condor-G/DAGMan

      • Job state “unfinished”

    • Monitor Job statistics

  • Output

    • Exit status (if available)

    • Final Job statistics (resource usage)

      • CPU

      • Storage

      • Bandwidth

    • Entity account update

    • Job state “remove”

Data Warehouse

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Submitter

Data Management

Information Gatherer


Data management components l.jpg

Data Management Components

Sphinx Server

  • “User Demanded Movement” Module

    • Input

      • Source(s)

      • Destination(s)

    • Output

      • DMC update

  • “Pre-Planned Movement” Module (most important to Sphinx)

    • Input

      • Available Data

      • Available Resources

      • Prediction of Data Access Patterns

      • DAG (“reduced”)

      • Jobs (“unsent”)

      • Job(s) schedule for next k steps

    • Output

      • Data Movement

      • DMC update

  • Prediction Module

    • Input:

      • Historical Data Access Patterns

    • Output:

      • Future Data Access Patterns

Data Warehouse

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Submitter

Data Management

Information Gatherer


Information gathering module l.jpg

Information Gathering Module

Sphinx Server

  • Resource Property Module

    • Input

      • Grid Information Systems (such as MDS using GLUE Schema, etc)

    • Output

      • Resource Property views

  • Job Tracking Module

    • Input

      • Job Monitoring information (such as Condor-G, etc)

    • Output

      • Work Views

  • Grid Weather Module

    • Input

      • Grid Monitoring systems (such as GEMS, MonALISA, etc)

    • Output

      • Grid Weather Views

        • CPU

        • Storage

        • Bandwidth

  • Grid Weather Prediction Module

    • Input:

      • Grid Weather History

    • Output:

      • Grid Weather Forcast for next k steps

Data Warehouse

Message Interface

DAG Reducer

Multi-threaded

Control Process

Prediction Engine

Admission Control

Planner

Submitter

Data Management

Information Gatherer


  • Login