Totalview multiprocess debugger
1 / 12

TotalView Multiprocess Debugger - PowerPoint PPT Presentation

  • Uploaded on

Matt Wolfe LC Development Environment Group Lawrence Livermore National Laboratory. TotalView Multiprocess Debugger. Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'TotalView Multiprocess Debugger' - min

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Totalview multiprocess debugger

Matt WolfeLC Development Environment GroupLawrence Livermore National Laboratory

TotalView Multiprocess Debugger

Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551

This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344


Totalview uses
TotalView Uses

  • Code correction

  • Code exploration and learning

  • Code statement runtime operation study

  • Memory utilization and debugging

  • Languages: C, C++, Fortran, UPC

  • Paradigms: serial, MPI, OpenMP, pthreads, CUDA (coming)

Totalview technologies tools
TotalView Technologies' Tools

  • Workbench: tvworkbench

  • MemoryScape: memscape, memscript

  • TotalView: totalview, totalviewcli, tvscript, visualize

  • ReplayEngine: totalview -replay

  • Remote Display Client:

  • contributed: colortv, mtv, mxterm, sxterm

When to use totalview tv
When To Use TotalView (TV)

  • Start code under TV

  • Attach to running code with TV

  • Post mortem core file inspection

Preparing code and invoking totalview
Preparing Code and Invoking TotalView

  • Compile: -g -O0

  • IBM: -qsaveopt OpenMP: -qsmp=omp:noopt

  • totalview -args <code command line>

  • totalview

  • totalview code corefile

Totalview basic tools
TotalView Basic Tools

  • Visual and one-click access to greater detail

  • Breakpoints, Barrier points, TV intrinsics

  • Program control, Src/Asm single-stepping, restart, detach

  • Source, stack trace, stack frame (args, locals, registers)

  • View/change data, tooltips, Last Value, change highlighting

  • File/text/symbol search

  • Thread control, signal control

Totalview power tools
TotalView Power Tools

  • Watchpoints, Conditional Breakpoints, Code Patching

  • Statistics, Filtering, Sorting, Visualization, Dump Format

  • Call Graph, Show Across, Custom Groups, Disassembler

  • Expression List, Evaluate, Type Transformation

  • Message Queue Display and Graph

  • Program Browser, Fortran Modules, Thread Objects

  • Memory Debugger, Replay Engine, Command Line Interface

  • tvscript

Working at scale
Working at Scale

  • Reduce your process count if possible


  • Subset attach

  • Tools > Event Log

  • Work up to your target scale and be patient

  • Processes and Threads tabs

  • Sort by process state and collapse for counts

  • Filter out uninteresting libraries' symbols

Tvtech resources
TVTech Resources


  • Help menu

  • F1 key

  • CLI help command


  • (FAQs, Q&A wisdom)

  • Your local development tools support team, Tri-Labs issue tracker

Afternoon session
Afternoon Session

  • Hear use cases

  • How to see shared object dependencies: file, ldd, cat /proc/<PID>/maps

  • How to trace system calls: man strace (x86-64); man truss (rs6000)

  • How to drop a useful core file: limit coredumpsize unlimited, ulimit -c unlimited, ^\

  • How to determine your TotalView version: -V

  • How to start with a clean slate: rm -f –r \ ~/.{memscape,preferences.tvd,totalview,tvworkbench} \ {~,.}/.tvdrc

  • How to get TV to display your source: $tree()

Afternoon session continued
Afternoon Session (Continued)

  • How to stop TV before the stack is unwound: ifort: for__issue_diagnostic

  • How to catch a Nan: CDWP(C, $double, 8 bytes): if ($is_nan($newval)) $stop

  • How to catch heap corruption

  • How to quickly return to a location of interest: $count

  • How to trace a variable (x86-64) printf("#"); fflush( (void *) 0);

  • How to display an array using an array of indices Ask Jeff <> about UPDD.

Afternoon session continued1
Afternoon Session (Continued)

  • How to present memory leaks to a library provider

  • How to check instantaneous memory changes (x86-64) printf("\nTV node ID: %d, PID: %d\n",$nid,$pid); malloc_stats()

  • How to reconstruct an incomplete stack trace

  • How to find missing subroutine arguments and locals

  • Beginning to use the CLI: help

  • Discovering how to get TV to accept an expression