SPEC Benchmarks for Large Systems
This presentation is the property of its rightful owner.
Sponsored Links
1 / 22

SPEC Benchmarks for Large Systems PowerPoint PPT Presentation


  • 79 Views
  • Uploaded on
  • Presentation posted in: General

SPEC Benchmarks for Large Systems Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann, R. Lieberman, M. van Waveren, and B. Whitney SPEC High Performance Group [email protected] What is SPEC?.

Download Presentation

SPEC Benchmarks for Large Systems

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


SPEC Benchmarks for Large Systems

Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann, R. Lieberman, M. van Waveren, and B. Whitney

SPEC High Performance Group

[email protected]


What is SPEC?

The Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops suites of benchmarks and also reviews and publishes submitted results from our member organizations and other benchmark licensees.


SPEC High-Performance Group

  • Founded in 1994

  • Mission: To establish, maintain, and endorse a suite of benchmarks that are representative of real-world high-performance computing applications.

  • SPEC/HPG includes members from both industry and academia.

  • Benchmark products:

    • SPEC OMP (OMPM2001, OMPL2001)

    • SPEC HPC2002 released at SC 2002


Currently active SPEC HPG Members

  • Fujitsu

  • HP

  • IBM

  • Intel

  • SGI

  • SUN

  • UNISYS

  • University of Purdue

  • University of Stuttgart


Where is SPEC Relative to Other Benchmarks ?

There are many metrics, each one has its purpose

Raw machine performance: Tflops

Microbenchmarks: Stream

Algorithmic benchmarks: Linpack

Compact Apps/Kernels:NAS benchmarks

Application Suites: SPEC

User-specific applications: Custom benchmarks


Comparison of different benchmark classes


SPEC OMP

  • Benchmark suite developed by SPEC HPG (High Performance Group)

  • Benchmark suite for performance testing of shared memory processor systems

  • Uses OpenMP versions of SPEC CPU2000 benchmarks and candidates


SPEC OMP/CPU2000 differences

  • Larger working set sizes, 1.6GB for OMPM2001, 6.5 GB for OMPL2001; it is 200MB for CPU2000

  • Longer run times (>1000 s/cpu for CPU2000 vs >10,000 s/cpu for OMP2000 medium)

  • Focus on SMP systems, and issued by HPG

  • SPEC OMP based on work for CPU2000, SPEC OMP mixes integer and FP in one suite

  • OMPM is focused on 4-way to 16-way systems

  • OMPL is targeting 32-way and larger systems


SPEC OMP Results

  • 66 submitted results for OMPM

  • 24 submitted results for OMPL


SPEC OMPL Results: Benchmark with scaling to 128


SPEC OMPL Results: Superlinear scaling of applu


SPEC OMPL Results: Benchmark with scaling to 64


SPEC HPC2002 Benchmark

  • Full Application benchmarks(including I/O) targeted at HPC platforms

  • Serial and parallel (OpenMP and/or MPI)

  • Currently three applications:

    • SPECenv: weather forecast

    • SPECseis: seismic processing, used in the search for oil and gas

    • SPECchem: comp. chemistry, used in chemical and pharmaceutical industries (gamess)

  • All codes include several data sizes


SPECenv execution models on a Sun Fire 6800

Medium scales better

OpenMP best for small size

MPI best for medium size


SPEC HPC2002 Results: SPECenv scaling


SPECseis execution models on a Sun Fire 6800

Medium scales better

OpenMP scales better than MPI


SPEC HPC2002 Results: SPECseis scaling


SPECchem execution models on a Sun Fire 6800

Medium shows better scalability

MPI is better than OpenMP


SPEC HPC2002 Results: SPECchem scaling


Hybrid Execution for SPECchem


Current and Future Work

  • SPEC HPC:

    • Update of SPECchem

    • Improving portability, including tools

    • Larger datasets

  • New release of SPEC OMP:

    • Inclusion of alternative sources

    • Merge OMPM and OMPL on one CD

  • Adoption of new benchmark codes:

    • Should represent a type of computation that is regularly performed on HPC systems

    • We currently examine CPU2004 for candidates


Conclusion and Summary

  • Results of OMPL and HPC2002:

    • Scalability of many programs to 128 CPUs

  • Best choice of programming model (MPI,OpenMP, hybrid) depends on:

    • Hardware

    • Program

    • Data set size

  • SPEC HPC will continue to update and improve the benchmark suites.


  • Login