Controlling process thread placement on zeus
Download
1 / 14

Controlling Process/Thread Placement on Zeus - PowerPoint PPT Presentation


  • 92 Views
  • Uploaded on

Controlling Process/Thread Placement on Zeus. Raghu Reddy ( [email protected] ). Placement example. GFS, T574 case, with 241 MPI ranks Nodes=121, PPN=2, NUMTHD=5 2 cores left idle Artificial example to illustrate the problem . Default or “basic” omplace.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Controlling Process/Thread Placement on Zeus' - cassidy-albert


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Controlling process thread placement on zeus

Controlling Process/ThreadPlacement on Zeus

Raghu Reddy ([email protected])

NCEP MTT Meeting


Placement example
Placement example

  • GFS, T574 case, with 241 MPI ranks

  • Nodes=121, PPN=2, NUMTHD=5

    • 2 cores left idle

  • Artificial example to illustrate the problem 

NCEP MTT Meeting


Default or basic omplace
Default or “basic” omplace

qsub -l nodes=121:ppn=2 jobfile

mpiexec_mpt -np 241 omplace –nt 5 ./a.out

NCEP MTT Meeting


One zeus node
One Zeus node

Node

Socket

C

C

C

C

C

C

C

C

C

C

C

C

  • Two Sockets

  • Each with it’s own memory (12 GB each, 24 GB Total)

  • QPI makes it possible for 1 sock to access the other’s memory

  • Non Uniform Memory Access (NUMA)

    • Different BW

    • Difference Latency

  • Memory BW is shared by 6 cores

NCEP MTT Meeting


Omplace with bs and st set
Omplace with bsand st set

NCEP MTT Meeting


Controlling placement
Controlling placement

  • bs = (ppn*numthds)/2

  • mpiexec_mpt-np 241 omplace–c 0-:bs=$bs+st=6 –nt5 ./a.out

-c CPU list

BS Block Size

ST Stride

NCEP MTT Meeting


Impact of proper placement
Impact of Proper Placement?

  • Depends on the application

    • If not memory BW limited, minimal impact

      • DGEMM

    • Memory BW limited applications will see good benefit

      • STREAM

    • Most applications fall in-between

NCEP MTT Meeting


Impact of placement kernels
Impact of Placement: Kernels

  • HPCC Benchmark

  • MPI benchmark to characterize performance

    • “Single” (we will ignore this for today)

    • “Star” – Independent copy per core

NCEP MTT Meeting


Controlling placement gfs
Controlling placement: GFS

  • bs = (ppn*numthds)/2

  • mpiexec_mpt-np 241 omplace–c 0-:bs=$bs+st=6 –nt5 ./a.out

  • fe2% dd 20:13:36 20:17:39 nt-5-ppn-2-noomplace

  • 243 nt-5-ppn-2-noomplace

  • fe2% dd 16:26:01 16:28:47 nt-5-ppn-2

  • 166 nt-5-ppn-2

  • fe2% dd 16:30:32 16:33:00 nt-5-ppn-2-bs-5

  • 148 nt-5-ppn-2-bs-5

  • fe2%

NCEP MTT Meeting


A more practical example
A More Practical example

  • Your MPI (non-threaded application) needs more memory than what is available per core

    • So you have to use ppn=6 instead of ppn=12

    • If you run it without omplace, all ranks would be put on 1 socket

    • 6 ranks on one socket, 0 ranks on the second!

    • Use bs = 3 (ppn*numthd/2)

    • This will put 3 ranks on each socket

    • Improves memory bandwidth

NCEP MTT Meeting


Test run of gfs no openmp
Test run of GFS (no OpenMP)

  • GFS, T574 case, with 241 MPI ranks

  • Nodes=41, PPN=6, NUMTHD=1

noomp runs

------------

fe2% dd 13:10:50 13:16:35 nt-1-ppn-6

345 nt-1-ppn-6

fe2% dd 13:26:55 13:32:13 nt-1-ppn-6-nt-2

318 nt-1-ppn-6-nt-2

fe2% dd 13:57:15 14:02:27 nt-1-ppn-6-bs-3

312 nt-1-ppn-6-bs-3

fe2%

NCEP MTT Meeting


Gfs nodes 41 ppn 6 thds 2
GFS:Nodes=41, PPN=6, THDS=2

fe2% dd 14:51:07 14:54:52 nt-2-ppn-6 (no TAU)

225 nt-2-ppn-6

fe2% dd 14:27:57 14:32:17 tau-3files-nt-2-ppn-6

260 tau-3files-nt-2-ppn-6

fe2%

NCEP MTT Meeting


Summary
Summary

  • Under certain circumstances using proper placement can be beneficial

  • In general, if you’re using all the available cores this may not be important.

  • This may be significant if you are leaving some cores idle where it may be beneficial.

    • Especially so, if you idle cores and use “remote” memory

NCEP MTT Meeting


Questions
Questions?

  • Thanks!

NCEP MTT Meeting


ad