random access to event traces with otf n.
Download
Skip this Video
Download Presentation
Random Access to Event Traces with OTF

Loading in 2 Seconds...

play fullscreen
1 / 24

Random Access to Event Traces with OTF - PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on

Center for Information Services and High Performance Computing (ZIH). Random Access to Event Traces with OTF. Dagstuhl Seminar N ° 07341 August 19th - 24th 2007. OUTLINE. INTRODUCTION OTF FEATURES Openness, Flexibility, Performance OTF Architecture AUXILIARY INFORMATION

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Random Access to Event Traces with OTF' - zeal


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
random access to event traces with otf

Center for Information Services and High Performance Computing (ZIH)

Random Access to Event Traces with OTF

Dagstuhl Seminar N°07341

August 19th - 24th 2007

outline
OUTLINE
  • INTRODUCTION
  • OTF FEATURES
      • Openness, Flexibility, Performance
      • OTF Architecture
  • AUXILIARY INFORMATION
  • STATISTIC RECORDS
      • What do we need it for?
  • SNAPSHOT RECORDS
      • Example of use
  • CONCLUSION

Heike Jagode

1 introduction
1. INTRODUCTION
  • INTRODUCTION
  • OTF FEATURES
      • Openness, Flexibility, Performance
      • OTF Architecture
  • AUXILIARY INFORMATION
  • STATISTIC RECORDS
      • What do we need it for?
  • SNAPSHOT RECORDS
      • Example of use
  • CONCLUSION

Heike Jagode

requirements
REQUIREMENTS
  • Development of scalable tracing tools for HPC platforms requires:
    • Low-overhead trace measurement system to generate trace data

AND

    • (Efficient trace analysis tools to process data)
  • Crucial factor to trace tool development is an open specification of trace information that provides:
    • Target for trace generation

AND

    • (enables trace analysis and visualisation tools to operate efficiently at large scale)
  • The Open Trace Format (OTF) is such a trace definition and representation for the use with large-scale parallel platforms

Heike Jagode

2 otf features
2. OTF FEATURES
  • INTRODUCTION
  • OTF FEATURES
      • Openness, Flexibility, Performance
      • OTF Architecture
  • AUXILIARY INFORMATION
  • STATISTIC RECORDS
      • What do we need it for?
  • SNAPSHOT RECORDS
      • Example of use
  • CONCLUSION

Heike Jagode

openness flexibility performance
OPENNESS, FLEXIBILITY, PERFORMANCE
  • Design of OTF is directed at 3 objectives:
  • Openness

open format defines record types and file structure so that OTF traces can be generated and read correctly

external wishes will be considered .. just talk to us!

  • Flexibility

efficiently selective access is supported

  • Performance

 is determined by how efficient & fast OTF trace query and manipulation can be done

 parallel I/O

Heike Jagode

selection of otf features
SELECTION OF OTF FEATURES
  • Supports fast and selective access to large amount of performance trace data
  • Based on a stream model  single separate units represent segments of the overall data
  • OTF streams may contain multiple independent processes whereas on process belongs to a single stream exclusively
  • Encourages parallel I/O
  • Strictly sequential reading of parallel traces still supported
  • Allows transparent ZLib compression

Heike Jagode

multiple streams
MULTIPLE STREAMS

Heike Jagode

3 auxiliary information
3. AUXILIARY INFORMATION
  • INTRODUCTION
  • OTF FEATURES
      • Openness, Flexibility, Performance
      • OTF Architecture
  • AUXILIARY INFORMATION
  • STATISTIC RECORDS
      • What do we need it for?
  • SNAPSHOT RECORDS
      • Example of use
  • CONCLUSION

Heike Jagode

auxiliary information
AUXILIARY INFORMATION
  • Usually, traces are read linearly from the beginning
  • OTF introduces possibility to access arbitrary time stamps fast
  • Some auxiliary information becomes necessary
  • STATISTIC RECORDS
    • Provide an overview over an entire interval of time
    • Point to certain sections that are worth to access
  • After finding section of interest, no need in reading everything before that certain time stamp
  • SNAPSHOT RECORDS
    • Collect current state of all participating processes to make it possible to start reading at the certain time stamp

Heike Jagode

4 statistic records
4. STATISTIC RECORDS
  • INTRODUCTION
  • OTF FEATURES
      • Openness, Flexibility, Performance
      • OTF Architecture
  • AUXILIARY INFORMATION
  • STATISTIC RECORDS
      • What do we need it for?
  • SNAPSHOT RECORDS
      • Example of use
  • CONCLUSION

Heike Jagode

statistic records 1 4
STATISTIC RECORDS 1/4
  • Statistic information about a monotonically increasing property p(t) for an interval [a,b) - result p([a,b)) can be computed as:

p([a,b)) = p([0,b)) - p([0,a))

  • Accumulate from the beginning of the trace until the current time stamp (e.g. exclusive time per function)
  • With n points in time t0, ..., tn-1, there are (n*(n-1))/2 possible interval results p([ti,tj)), i ≠ j of varying granularity
  • Quick overview over whole trace
  • Without reading all events (huge)
  • Read special statistic records only (small)

Heike Jagode

statistic records 3 4
STATISTIC RECORDS 3/4
  • For Function Calls statistics involve:
    • Number of calls
    • Exclusive / Inclusive Time per function and function group
  • For Point-to-Point and Collective Communications, statistics provide summarized information for a given message type:
    • Process where message originated
    • Peer - process where message is sent to
    • Communicator of message summary / message type / tag
    • Number of sent and received messages
    • Number of bytes sent via messages of the given type
    • Number of bytes received through messages of the given type

Heike Jagode

statistic records 4 4
STATISTIC RECORDS 4/4
  • Statistics provide summarized information about File Operations:
    • File identifier (or 0)
    • Process where file operations occurred
    • Number of open events / Number of close events
    • Number of read events / Number of write events
    • Number of seek events
    • Number of bytes read
    • Number of bytes written
  • Same applies to File Operations in a File Group

Heike Jagode

5 snapshot records
5. SNAPSHOT RECORDS
  • INTRODUCTION
  • OTF FEATURES
      • Openness, Flexibility, Performance
      • OTF Architecture
  • AUXILIARY INFORMATION
  • STATISTIC RECORDS
      • What do we need it for?
  • SNAPSHOT RECORDS
      • Example of use
  • CONCLUSION

Heike Jagode

snapshot records 1 5
SNAPSHOT RECORDS 1/5
  • After analyzing overview information that points to a certain section of interest
  • Snapshot Records allow loading this section instead of entire trace
  • In order to start reading from a certain time stamp, the current state of ALL participating processes needs to be known
  • The Snapshot Recordsexplicitly store this information
  • In detail it means, snapshots provide:
    • The call stack (i.e. all active function calls)
    • List of pending messages, ongoing I/O activities
    • Current OpenMP regions, etc ... at a point in time
  • Based on this information, start reading event records at that very time stamp

Heike Jagode

snapshot records 2 5
SNAPSHOT RECORDS 2/5

Heike Jagode

snapshot records 4 5
SNAPSHOT RECORDS 4/5
  • For Function Calls snapshots provide information about:
    • a past function call at the time “original time“
    • Function which has been entered
    • Process where action took place
    • Explicit source code location identifier > 0 (or 0)
  • Information about a past message send operation
    • at the time “original time“
    • Sender and Receiver of the message
    • Process-group to which sender and receiver belong to (or 0)
    • Message type information > 0 (or 0) and message length
    • Explicit source code location identifier > 0 (or 0)

Heike Jagode

snapshot records 5 5
SNAPSHOT RECORDS 5/5
  • Provide a snapshot record for opened (and not yet closed) files
    • Timestamp when the file has been opened
    • Process identifier
    • Unique file open identifier

Heike Jagode

6 conclusion
6. CONCLUSION
  • INTRODUCTION
  • OTF FEATURES
      • Openness, Flexibility, Performance
      • OTF Architecture
  • AUXILIARY INFORMATION
  • STATISTIC RECORDS
      • What do we need it for?
  • SNAPSHOT RECORDS
      • Example of use
  • CONCLUSION

Heike Jagode

conclusion
CONCLUSION
  • Beside the stream model which encourages parallel I/O:
  • The Statistic Records give a quick overview over the whole trace
    • Without reading all events (huge)
    • Point to certain sections worth accessing
  • The Snapshot Records collect current state of all participating processes
    • To make it possible to start reading at the certain time stamp
    • Just load sections of interest instead of entire huge trace

Heike Jagode

contact details
CONTACT DETAILS
  • It is still at an early stage
  • You are very welcome to send wishes to either:
    • Andreas Knüpfer  he is your man
    • andreas.knuepfer@tu-dresden.de
    • Or myself
    • heike.jagode@tu-dresden.de

Heike Jagode

ad