SPEC Benchmarks for Large Systems
This presentation is the property of its rightful owner.
Sponsored Links
1 / 22

SPEC Benchmarks for Large Systems PowerPoint PPT Presentation


  • 76 Views
  • Uploaded on
  • Presentation posted in: General

SPEC Benchmarks for Large Systems Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann, R. Lieberman, M. van Waveren, and B. Whitney SPEC High Performance Group [email protected] What is SPEC?.

Download Presentation

SPEC Benchmarks for Large Systems

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Spec benchmarks for large systems

SPEC Benchmarks for Large Systems

Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann, R. Lieberman, M. van Waveren, and B. Whitney

SPEC High Performance Group

[email protected]


What is spec

What is SPEC?

The Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops suites of benchmarks and also reviews and publishes submitted results from our member organizations and other benchmark licensees.


Spec high performance group

SPEC High-Performance Group

  • Founded in 1994

  • Mission: To establish, maintain, and endorse a suite of benchmarks that are representative of real-world high-performance computing applications.

  • SPEC/HPG includes members from both industry and academia.

  • Benchmark products:

    • SPEC OMP (OMPM2001, OMPL2001)

    • SPEC HPC2002 released at SC 2002


Currently active spec hpg members

Currently active SPEC HPG Members

  • Fujitsu

  • HP

  • IBM

  • Intel

  • SGI

  • SUN

  • UNISYS

  • University of Purdue

  • University of Stuttgart


Where is spec relative to other benchmarks

Where is SPEC Relative to Other Benchmarks ?

There are many metrics, each one has its purpose

Raw machine performance: Tflops

Microbenchmarks: Stream

Algorithmic benchmarks: Linpack

Compact Apps/Kernels:NAS benchmarks

Application Suites: SPEC

User-specific applications: Custom benchmarks


Comparison of different benchmark classes

Comparison of different benchmark classes


Spec benchmarks for large systems

SPEC OMP

  • Benchmark suite developed by SPEC HPG (High Performance Group)

  • Benchmark suite for performance testing of shared memory processor systems

  • Uses OpenMP versions of SPEC CPU2000 benchmarks and candidates


Spec omp cpu2000 differences

SPEC OMP/CPU2000 differences

  • Larger working set sizes, 1.6GB for OMPM2001, 6.5 GB for OMPL2001; it is 200MB for CPU2000

  • Longer run times (>1000 s/cpu for CPU2000 vs >10,000 s/cpu for OMP2000 medium)

  • Focus on SMP systems, and issued by HPG

  • SPEC OMP based on work for CPU2000, SPEC OMP mixes integer and FP in one suite

  • OMPM is focused on 4-way to 16-way systems

  • OMPL is targeting 32-way and larger systems


Spec omp results

SPEC OMP Results

  • 66 submitted results for OMPM

  • 24 submitted results for OMPL


Spec ompl results benchmark with scaling to 128

SPEC OMPL Results: Benchmark with scaling to 128


Spec ompl results superlinear scaling of applu

SPEC OMPL Results: Superlinear scaling of applu


Spec ompl results benchmark with scaling to 64

SPEC OMPL Results: Benchmark with scaling to 64


Spec hpc2002 benchmark

SPEC HPC2002 Benchmark

  • Full Application benchmarks(including I/O) targeted at HPC platforms

  • Serial and parallel (OpenMP and/or MPI)

  • Currently three applications:

    • SPECenv: weather forecast

    • SPECseis: seismic processing, used in the search for oil and gas

    • SPECchem: comp. chemistry, used in chemical and pharmaceutical industries (gamess)

  • All codes include several data sizes


Specenv execution models on a sun fire 6800

SPECenv execution models on a Sun Fire 6800

Medium scales better

OpenMP best for small size

MPI best for medium size


Spec hpc2002 results specenv scaling

SPEC HPC2002 Results: SPECenv scaling


Specseis execution models on a sun fire 6800

SPECseis execution models on a Sun Fire 6800

Medium scales better

OpenMP scales better than MPI


Spec hpc2002 results specseis scaling

SPEC HPC2002 Results: SPECseis scaling


Specchem execution models on a sun fire 6800

SPECchem execution models on a Sun Fire 6800

Medium shows better scalability

MPI is better than OpenMP


Spec hpc2002 results specchem scaling

SPEC HPC2002 Results: SPECchem scaling


Hybrid execution for specchem

Hybrid Execution for SPECchem


Current and future work

Current and Future Work

  • SPEC HPC:

    • Update of SPECchem

    • Improving portability, including tools

    • Larger datasets

  • New release of SPEC OMP:

    • Inclusion of alternative sources

    • Merge OMPM and OMPL on one CD

  • Adoption of new benchmark codes:

    • Should represent a type of computation that is regularly performed on HPC systems

    • We currently examine CPU2004 for candidates


Conclusion and summary

Conclusion and Summary

  • Results of OMPL and HPC2002:

    • Scalability of many programs to 128 CPUs

  • Best choice of programming model (MPI,OpenMP, hybrid) depends on:

    • Hardware

    • Program

    • Data set size

  • SPEC HPC will continue to update and improve the benchmark suites.


  • Login