Access patterns metadata and performance
This presentation is the property of its rightful owner.
Sponsored Links
1 / 14

Access Patterns, Metadata, and Performance PowerPoint PPT Presentation


  • 48 Views
  • Uploaded on
  • Presentation posted in: General

Access Patterns, Metadata, and Performance. Alok Choudhary and Wei-Keng Liao Department of ECE, Northwestern University Collaboration with ANL. SDM kickoff meeting July 10-11, 2001. Virtuous Cycle. Simulation (Execute app, Generate data). Problem setup (Mesh, domain Decomposition).

Download Presentation

Access Patterns, Metadata, and Performance

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Access patterns metadata and performance

Access Patterns, Metadata, and Performance

Alok Choudhary and Wei-Keng Liao

Department of ECE, Northwestern University

Collaboration with ANL

SDM kickoff meeting

July 10-11, 2001

[email protected] 1


Virtuous cycle

Virtuous Cycle

Simulation

(Execute app,

Generate data)

Problem setup

(Mesh, domain

Decomposition)

Manage,

Visualize, Analyze

Measure Results,

Learn, Archive

[email protected] 2


Data access sequence dependency

Data Access Sequence Dependency

  • Temporal dependency

    • Access the same data set at different time stamp

  • Spatial dependency

    • Access different data sets at the same time stamp

  • Resolution dependency

    • Access the same data set at different resolution

  • Sequence is useful for I/O performance improvement, eg. Pre-fetch, pre-stage, storage continuity

[email protected] 4


Spatial data access patterns

Spatial Data Access Patterns

  • Parallel partition patterns:

    • Regular, irregular

    • Static, dynamic during simulation

  • Access sequence

    • Spatial, temporal, resolution

  • Access frequency

    • Once only, multiple times (overwrite for restart)

  • Access amount

    • Large, medium, small chunks

[email protected] 5


Access patterns for visualization analysis

Access Patterns for Visualization/Analysis

  • Generated from real data during simulation or in post-simulation process

  • Smaller size than real data

    • Type conversion,

      eg. float ï‚®unsign char

    • Reduce/increase resolution

    • Projection 3D to 2D

  • 3 types of data generate and display sequence

[email protected] 6


Architecture

Architecture

Simulation

Data Analysis

Visualization

User

Applications

I/O func (best_I/O (for these param))

Hint

Query

Input Metadata

Hints, Directives

Associations

Data

OIDs

parameters for I/O

Schedule, Prefetch, cache

Hints (coll I/O)

Storage Systems

(I/O Interface)

MDMS

Performance Input

System metadata

Metadata

access pattern, history

MPI-IO

(Other interfaces..)

[email protected] 7


Approach

Approach

  • Management meta data using OR-DBMS

    • Collect and organize meta data in relation tables

    • Design meta data query interface using SQL

  • Access to HSS

    • Obtain current storage layout, configuration

    • Native I/O interfaces or MPI-IO

  • I/O optimization

    • Determine optimal I/O calls

    • Overlap I/O with computation, communication, and I/O

    • Pre-fetch, pre-stage, migrate, purge in HSS

    • Sub-filing for large file, file container for small files

[email protected] 8


Metadata

Metadata

  • Application Level

    • Algorithms, compiling, execution environments

    • Time stamps, parameters, result summary

  • Programming Level

    • Data types, structures, association of datasets, partition patterns

  • Storage System Level

    • File locations, file structure, I/O modes, host names, device types, path names, storage hierarchy

  • Performance Level

    • I/O bandwidth of HSS for local and remote access

    • Data access sequence, frequency, other access hints

    • Collective or non-collection I/O

[email protected] 10


Applications

Applications

  • Asto3D -- study the highly turbulent convective

    layers of late-type star

    • Write only

    • regular partition on all data sets

  • ENZO -- simulate the formation of a cluster of

    galaxies consisting of gas and stars

    • Both read and write

    • Both regular and irregular partition

    • Adaptive Mesh Refinement dynamic load balancing

  • Common feature

    • Checkpoint / restart

    • Post-simulation data analysis

    • Visualizing the process of the computation in the form of a movie

[email protected] 11


Interface

Interface

[email protected] 12


Run application

Run Application

[email protected] 13


Dataset and access pattern table

Dataset and Access Pattern Table

[email protected] 14


Data analysis

Data Analysis

[email protected] 15


Integrating analysis

Integrating Analysis

Simulation

(Execute app,

Generate data)

On-line analysis

And mining

Problem setup

(Mesh, domain

Decomposition)

Manage,

Visualize, Analyze

Measure Results,

Learn, Archive

[email protected] 16


  • Login