access patterns metadata and performance
Download
Skip this Video
Download Presentation
Access Patterns, Metadata, and Performance

Loading in 2 Seconds...

play fullscreen
1 / 14

Access Patterns, Metadata, and Performance - PowerPoint PPT Presentation


  • 71 Views
  • Uploaded on

Access Patterns, Metadata, and Performance. Alok Choudhary and Wei-Keng Liao Department of ECE, Northwestern University Collaboration with ANL. SDM kickoff meeting July 10-11, 2001. Virtuous Cycle. Simulation (Execute app, Generate data). Problem setup (Mesh, domain Decomposition).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Access Patterns, Metadata, and Performance' - linus


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
access patterns metadata and performance

Access Patterns, Metadata, and Performance

Alok Choudhary and Wei-Keng Liao

Department of ECE, Northwestern University

Collaboration with ANL

SDM kickoff meeting

July 10-11, 2001

[email protected] 1

virtuous cycle
Virtuous Cycle

Simulation

(Execute app,

Generate data)

Problem setup

(Mesh, domain

Decomposition)

Manage,

Visualize, Analyze

Measure Results,

Learn, Archive

[email protected] 2

data access sequence dependency
Data Access Sequence Dependency
  • Temporal dependency
    • Access the same data set at different time stamp
  • Spatial dependency
    • Access different data sets at the same time stamp
  • Resolution dependency
    • Access the same data set at different resolution
  • Sequence is useful for I/O performance improvement, eg. Pre-fetch, pre-stage, storage continuity

[email protected] 4

spatial data access patterns
Spatial Data Access Patterns
  • Parallel partition patterns:
    • Regular, irregular
    • Static, dynamic during simulation
  • Access sequence
    • Spatial, temporal, resolution
  • Access frequency
    • Once only, multiple times (overwrite for restart)
  • Access amount
    • Large, medium, small chunks

[email protected] 5

access patterns for visualization analysis
Access Patterns for Visualization/Analysis
  • Generated from real data during simulation or in post-simulation process
  • Smaller size than real data
    • Type conversion,

eg. float unsign char

    • Reduce/increase resolution
    • Projection 3D to 2D
  • 3 types of data generate and display sequence

[email protected] 6

architecture
Architecture

Simulation

Data Analysis

Visualization

User

Applications

I/O func (best_I/O (for these param))

Hint

Query

Input Metadata

Hints, Directives

Associations

Data

OIDs

parameters for I/O

Schedule, Prefetch, cache

Hints (coll I/O)

Storage Systems

(I/O Interface)

MDMS

Performance Input

System metadata

Metadata

access pattern, history

MPI-IO

(Other interfaces..)

[email protected] 7

approach
Approach
  • Management meta data using OR-DBMS
    • Collect and organize meta data in relation tables
    • Design meta data query interface using SQL
  • Access to HSS
    • Obtain current storage layout, configuration
    • Native I/O interfaces or MPI-IO
  • I/O optimization
    • Determine optimal I/O calls
    • Overlap I/O with computation, communication, and I/O
    • Pre-fetch, pre-stage, migrate, purge in HSS
    • Sub-filing for large file, file container for small files

[email protected] 8

metadata
Metadata
  • Application Level
    • Algorithms, compiling, execution environments
    • Time stamps, parameters, result summary
  • Programming Level
    • Data types, structures, association of datasets, partition patterns
  • Storage System Level
    • File locations, file structure, I/O modes, host names, device types, path names, storage hierarchy
  • Performance Level
    • I/O bandwidth of HSS for local and remote access
    • Data access sequence, frequency, other access hints
    • Collective or non-collection I/O

[email protected] 10

applications
Applications
  • Asto3D -- study the highly turbulent convective

layers of late-type star

    • Write only
    • regular partition on all data sets
  • ENZO -- simulate the formation of a cluster of

galaxies consisting of gas and stars

    • Both read and write
    • Both regular and irregular partition
    • Adaptive Mesh Refinement dynamic load balancing
  • Common feature
    • Checkpoint / restart
    • Post-simulation data analysis
    • Visualizing the process of the computation in the form of a movie

[email protected] 11

integrating analysis
Integrating Analysis

Simulation

(Execute app,

Generate data)

On-line analysis

And mining

Problem setup

(Mesh, domain

Decomposition)

Manage,

Visualize, Analyze

Measure Results,

Learn, Archive

[email protected] 16

ad