D istributed i nteractive c omputing e nvironment
This presentation is the property of its rightful owner.
Sponsored Links
1 / 27

D istributed I nteractive C omputing E nvironment PowerPoint PPT Presentation


  • 51 Views
  • Uploaded on
  • Presentation posted in: General

D istributed I nteractive C omputing E nvironment. http:// www.arl.hpc.mil/SciVis/dice. Platforms. Challenge : Stovepiped Applications. HPC Codes. CFD Code 1. CSM Code 1. CCM Code 2. CCM Code 1. Preprocessing. Runtime Analysis. Tools. Postprocessing. Additional Challenges.

Download Presentation

D istributed I nteractive C omputing E nvironment

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


D istributed i nteractive c omputing e nvironment

Distributed

Interactive

Computing

Environment

http://www.arl.hpc.mil/SciVis/dice


D istributed i nteractive c omputing e nvironment

Platforms

Challenge : Stovepiped Applications

HPC Codes

CFD Code 1

CSM Code 1

CCM Code 2

CCM Code 1

Preprocessing

Runtime Analysis

Tools

Postprocessing


D istributed i nteractive c omputing e nvironment

Additional Challenges

  • Code Setup can be Difficult

  • Enormous Amounts of Data

  • Heterogeneous Environment

  • Security


D istributed i nteractive c omputing e nvironment

Distributed Interactive Computing Environment

Graphical User

Interface and Runtime Visualization

Common Distributed Data Model and Format

High Performance

Computing Resource

User’s Workstation


D istributed i nteractive c omputing e nvironment

Fully Integrated FMD GUI


D istributed i nteractive c omputing e nvironment

Fully Integrated ZnsFlow GUI


D istributed i nteractive c omputing e nvironment

Fully Integrated CTH GUI


D istributed i nteractive c omputing e nvironment

What is DICE ?

A Distributed Computing Environment Composed of Configurable Mega-Components

  • Heterogeneous DSM *

  • Data Organization

  • Graphical User Interface

  • Visualization

User-Ready Computing Environments Customized for Specific Technology Areas


D istributed i nteractive c omputing e nvironment

Network Distributed Global Memory

Clients

VectorPut VectorGet

Put Get

Barrier Semaphore

0

10M

30M

50M

Servers

Process Local Address Space

Disk File

System Shared Memory

Physical Data Storage


D istributed i nteractive c omputing e nvironment

Light Data

Heavy Data

  • 3D Structured Mesh

  • 300 x 200 x 300

  • Number of Materials

  • Material Names

  • Total Residual Mass

  • X, Y, Z Values

  • Material Volume Fractions

  • Cell Temperatures


D istributed i nteractive c omputing e nvironment

User Application

Scripting and Graphical User Interface Tools

Visual Generators

Plots

Planes

Surfaces

High Performance

Computing Code

Visualization

Toolkit (vtk)

DICE Object Directory

Object Oriented Support for Data Model, XML, ...

Hierarchical Data Format ( HDF5 )

Data Format : Number Type, Data Space ...

OpenGL

Network Distributed Global Memory ( NDGM )

with GSSAPI

Mesa

File I/O

Process Control

Shared Memory Facility

TCP/IP Sockets

Operating System

Distributed Resources


D istributed i nteractive c omputing e nvironment

DICE Data Model and Format

Convenience Layer

Structured Data Unstructured Data Groups

DICE Object Directory

Object Oriented Support for Data Model, XML, ...

Hierarchical Data Format ( HDF5 )

Data Format : Number Type, Data Space ...

FileSystem Serial and Parallel

Global Access to Secondary Storage ( Globus - GASS )

Network Distributed Global Memorywith GSSAPI

Other ...

Distributed Buffer of Bytes


D istributed i nteractive c omputing e nvironment

Hierarchical Data Format Version 5

Group

Attribute

Group

Group

DataSet

Attribute

Not Allowed

Group

DataSet

  • Host Independent Data Format

  • Access via API only

  • Physical Storage thru Drivers

  • Includes Data Compression

  • Includes SubSelections

Group

DataSet

DataSet

Attribute

Attribute

Attribute


D istributed i nteractive c omputing e nvironment

eXtensible Markup Language

  • Similar to HTML Format

  • Human Readable

  • Pervasive on the WEB

  • Strict Hierarchy with No Inheritance

  • Many Parsing Tools Already Exist

  • < Parent Name=“Top”>

    • < Child Name=“Left”>

      • Character Data for Left

    • < /Child >

    • <Child Name=“Right” >

      • Character Data for Right

    • < /Child >

  • < /Parent >

Parent

Child

Character Data for Left

Child

Character Data for Right

Parse

XML

Document Object Model

In-Memory Data Structures

Serialize


D istributed i nteractive c omputing e nvironment

N

5

3

3

5

4

4

3

1

N

4

3

N

3

1

2

2

2

1

2

2

1

1

4

Polyvertex

Polyline

Triangle

Quadrilateral

Polygon

6

4

5

5

2

3

3

4

1

3

1

4

1

2

2

Tetrahedron

Pyramid

Wedge

8

7

5

6

3

j

j

4

k

1

2

i

i

Hexahedron

2DSMesh

3DSMesh

DICE Object Directory Base Topologies


D istributed i nteractive c omputing e nvironment

Light Data

  • <Topology Type=“Tetrahedra” NumElements=“3”>

  • </Topology>

  • <Geometry>

    • <Vertices>

      • <Index Location=“XML” Type=“List”>

      • 1000 1001 21001 21000 ……

      • </Index>

      • <Points Type=“XYZ” Location=“HDF” >

      • NDGM:File1.h5:/Grid1/Geometry/MyPoints

      • </Points>

    • </Vertices>

  • </Geometry>

  • <Topology Type=“3DSmesh”

  • Dimensions=“10 20 30”>

  • </Topology>

  • <Geometry>

    • <Points Type=“XYZ” Location=“HDF” >

    • NDGM:File1.h5:/Grid1/Geometry/MyPoints

    • <Region Type=“Slab” Location=“XML” >

    • 1 2 10 2 3 20 3 4 30

    • </Region>

    • </Points>

  • </Geometry>

100 x 200 x 300

HPC Code

HDF5

. . . .

NDGM

GASS

MPI-I/O


D istributed i nteractive c omputing e nvironment

Complexity of Model Defined in Light Data

Domain

Domain

Grid

XML

Grid

Attribute

NDGM:Pressure.h5

Attribute

NDGM:Temperature.h5

Topology

100 Hexahedra

Geometry XYZPoints.h5

HPC Code

HDF5

HDF5

File

NDGM

GASS

HPC Code I/O is Natural and Efficient


D istributed i nteractive c omputing e nvironment

XML Contains Parameters for Variable Substitution

Domain

Domain

Grid

XML

Grid

  • Parameter TimeStep = 0 to 99

    • Attribute

    • NDGM:&TimeStep;/Pressure.h5

Topology

100 Hexahedra

Geometry XYZPoints.h5

HDF5

HDF5 for TimeStep 0

HPC Code

HDF5 for TimeStep 1

HDF5 for TimeStep 2


D istributed i nteractive c omputing e nvironment

Parallel / Distributed Entities

Post-Processing Tools

Pre-Processing Tools

Runtime Tools

HPC Codes

Convenience Layer

eXtensible Markup Language ( XML )

Hierarchical Data Format Verision 5 (HDF5 )

Distributed Data Hub


D istributed i nteractive c omputing e nvironment

DICE Data Reader for EnSight

Data Reader uses DICE Object Directory API

EnSight

DICE Object Directory


D istributed i nteractive c omputing e nvironment

DICE Visualization using vtk

HDF5

HDF5

DICE Object Directory

File

NDGM

GASS

Visualization “Subnet”

Source

Filter

Filter

Filter

Polygons


D istributed i nteractive c omputing e nvironment

Adding Performance Friendly Functionality

System Programming Language

C C++ Fortran77 Fortran90 etc.

SWIG

Compiled Object

Tcl Wrapper

}

New Scripting Command

via

Dynamically Loadable Library

Perl Wrapper

Python Wrapper

Java Wrapper


D istributed i nteractive c omputing e nvironment

C++

DiceFloat64ArrayMyData, NewData;

DiceInt64ArrayMyIndex;

MyData.SetNumberOfElements( 10000 );

MyData.Generate(0.0, .9999 );

MyIndex.SetNumberOfElements( 100 );

MyIndex.Generate( 0, 99 );

MyIndex *= 2;

NewData = MyData[ MyIndex ];

Allocate 10,000 Doubles

Set Individual Values

Array Operation +, -, *, /

Array Indexing

Scripting ( Tcl, Perl, etc. )

DiceHDF H5Object

H5Object Open NDGM:FileName.h5:/DataSetName

DiceMemory MemoryObject

MemoryObject SetArray [ DiceFloat32Array ]

H5Object Read [ MemoryObject cget -this ]

set NewData [ MemoryObject GetArray ]

DiceExpr $NewData[ where( $NewData > .5 ) ] = .5

Object Oriented Scripting Interface

Open HDF5 File, Query Number Type and Shape

Request Automatic Conversion to Float32

MemoryObject has Number Type and Shape

Clip Values to .5


D istributed i nteractive c omputing e nvironment

DICE Web Interface

Under Construction

DICE on HPC Resource

using Off Screen Rendering

SSL

Kerberos kinit

  • Every URL Request is Intercepted to Verify Session Id

  • DICE Runs as a User Process

  • HTML FORMS result in Tcl Procedure Invocation

  • Graphics Updated via Server-Push ( Netscape only ) or Client-Pull

  • Intended for Limited Interfaces from Remote ( Slow Connection ) or Graphically Challenged Client Machines.


D istributed i nteractive c omputing e nvironment

MPI Finite Element

MPI Finite Volume

Visualization

Data Mining

Steering

Coupling

HDF5

NDGM


D istributed i nteractive c omputing e nvironment

“Gently Coupling” Codes from Separate Disciplines

Distributed Data Hub

Finite Volume Structural Mechanics

Small Scale Analysis

Global Monitor

Injection Flow Analysis

Runtime Visualization


D istributed i nteractive c omputing e nvironment

COTS, GOTS and “Semi-Automatic” Visualization

DICE and the HPC Environment

X / OpenGL

DICE User Interface

XML and Http

( SSL + Kerberos )

DICE Server

( User Process )

DICE Object Directory

XML, HDF5 I/O

One or More Cooperating HPC Codes


  • Login