1 / 35

UKQCD software for lattice QCD

UKQCD software for lattice QCD. P.A. Boyle, R.D. Kenway and C.M. Maynard UKQCD collaboration. Contents. Motivation Brief introduction to QCD What is the science What we actually calculate BYOC UKQCD software Why use more than one code base UKQCD contributions to code bases Conclusions.

dianthe
Download Presentation

UKQCD software for lattice QCD

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. UKQCD software for lattice QCD P.A. Boyle, R.D. Kenway and C.M. MaynardUKQCD collaboration

  2. Contents • Motivation • Brief introduction to QCD • What is the science • What we actually calculate • BYOC • UKQCD software • Why use more than one code base • UKQCD contributions to code bases • Conclusions AHM2008

  3. What is stuff? Experiments similar to the Large Hadron Collider (LHC) probe the structure of matter Need a theory to interpret and understand phenomena and predictnew ones! LHC will switch on 10 September 2008 AHM2008

  4. The Structure of matter • Quarks have • Mass feel gravity • Charge  feel electromagnetism • Flavour  feel weak interaction • Colour  feel strong interaction • Strong interaction binds quarks into hadrons • Protons and neutron • Glued together by gluons AHM2008

  5. What are Gluons? Gluons carryor mediate the strong interaction Quarks feel each other’s presence by exchanging momentum via gluons (virtual spring?) Similar to photon in electromagnetism Unlike photon carry charge of strong interaction (colour) – couple to themselves Gluons are sticky! AHM2008

  6. Introducing QCD! • 1972 DJ. Gross, F. Wilzcek HD Politzer • construct a Quantum field theory of quarks and gluon based on a symmetry group for colour – Quantum Chromodynamics (QCD) • (prove QCD is asymptotically free) • QFT for strong interaction • 2004 Receive Noble prize • Are we done? … um, not quite AHM2008

  7. Asymptotic freedom • Short distance/high momentum strength of interaction is small • Converse is infrared slavery • Low momentum  strong coupling • Quarks are confined in hadrons • Proton mass is ~1 GeV A Feynman diagram • Analytic tool (perturbation theory) only works for small interactions AHM2008

  8. Quarks and gluons on a lattice • Replace 4d space-time with grid • Lattice spacing a • Quark fields on sites y(x) • 4 component spinor (complex vector) on each site • Gluon fields on links Um(x) • 3x3 complex matrix on each link • Equations of motion are partial differential equations • Replace with finite difference • Large ( Volume ), sparse matrix (Fermion matrix) • Contains quark-gluon coupling AHM2008

  9. Numerical computation • Infinite dimensional path integral  high dimensional sum • Hybrid Monte Carlo (HMC) and variants • Update quark and gluon fields • Invert the fermion matrix each update • Krylov subspace methods – conjugate gradient • Generate many paths – many gluon field configurations • Compute (or measure) quantities of interest on each configuration • Invert the fermion matrix • Average over all configurations AHM2008

  10. Why lattice QCD is hard • Fermion matrix is badly conditioned • up and down quarks are nearly massless • Statistical uncertainty • Require at least O(105) MC updates •  N~O(102) 1-5% stat error for basic quantities • Systematic uncertainty • Several quark masses (chiral limit) • Bigger box required for lighter masses • 2or more volumes (105-7) and lattice spacings • Scales badly with problem size • a6or a7 and at least 1/mq AHM2008

  11. The bottom line • Need to invert matrices which are • Very Large ~ O(107), • Really badly conditioned • CN  O(104) or more • Many, many times • ~O(106) AHM2008

  12. Quarks and gluons on a computer • Interaction is local • Nearest neighbour interactions Parallel decomposition Sub-volume of lattice on each processor Simple communication pattern (halo-swap) Regular communication pattern AHM2008

  13. Anatomy of a calculation • Gluon and quark fields distributed • Not Fermion matrix • Exploit local interactions (sparsity) when evaluating matrix-vector operations • Matrix-vector is M(x,y;U)•y(x) • Colour matrix U(x) is small and dense, not split across PE • Dominated by matrix-vector and global sums in iterative solver • Double-precision floating point • Computation versus communication • Smaller local volume • Greater proportion of data “near” processor  • More communication  • Machine limiting factors • Memory bandwidth • Comms latency and bandwidth • QCD is ideally suited to MPP machine • Build yer own? AHM2008

  14. QCD-on-a-chip (14K ASIC) • ASIC from IBM technology library • PowerPC 440 embedded CPU core • 64-bit FPU - One FMA per cycle • 4MB fast embedded DRAM On chip memory and Ethernet controller • Custom design • High speed serial links (DMA) • Prefetching EDRAM controller • Bootable Ethernet JTAG interface • 400 MHz  peak is 0.8Gflop/s • Network is 6d torus of nearest neighbour AHM2008

  15. QCDOC performance Saturate single link bandwidth for even small packet size  Low latency  Good for small local volume Global vol 164 22x42 local volume 1K PE Super-linearscaling as data goes “on-chip Linear thereafter AHM2008

  16. UKQCD collaboration • 8 UK universities • Plymouth joined in 2007 • Prior to QCDOC era (up to 2002) • Consensus on form of calculation • Collaboration owned FORTRAN code • Assembler kernel for performance on Cray T3D/T3E • QCDOC era • Several (3) different calculations • Each sub-group collaborates internationally • Two open source c++ codes • CPS and chroma • Assembler kernels for performance AHM2008

  17. SciDAC • US DoE program • Funds all US groups • Hardware and software • USQCD • Code development • Common code environment • UKQCD actively collaborates with USQCD • Sub-project by sub-project • USQCD and UKQCD orgs are funding based • Lateral collaboration based on science! • Collaborate on software module development AHM2008

  18. CPS before QCDOC • Developed by Columbia University (CU) for QCDSP machine • Ancestor of QCDOC • Originally not ANSI c++ code • many QCDSP specific features • Not readily portable • Belongs to CU developers • UKQCD chose this code base • Building your own machine is a big risk • CPS code base most likely to run on QCDOC from day 1 • Does have required functionality • EPCC project to ANSI-fy the code • Code now ran correctly (if slowly) everywhere else AHM2008

  19. UKQCD contribution to CPS • Assembler version of key kernel • P.A. Boyle via BAGEL assembler generator (see later) • UKQCD develops new Rational Hybrid Monte Carlo (RHMC) algorithm • Implement and test in CPS (Clarke and Kennedy) • New algorithm has many parameters • Tuning and testing is a non-trivial task • CU+BNL+RBRC (RBC) + UKQCD new physics project • (2+1 flavour DWF) • up and downdegenerate + strange quarks • UKQCD contribute to AsqTad 2+1 flavour project • Other contributors in USA (MILC) AHM2008

  20. RHMC • HMC alg evolves 2 degenerate flavours(M is fermion matrix) • Quark fields are anti-commuting Grassmann variables • Take square root to do one flavour • Approximate square root to a Rational Function • Roots andpolescalculated with multi-shift solver • Terms with largest contribution to the fermion force • Change the MC update the most • Cost the least to compute • Change CG tolerance • Loosen CG for terms which contribute least • Can reduce CG count 2x • Keep algorithm exactwith Metropolis accept/reject AHM2008

  21. Implementing multi-timescale RHMC • Can use RHMC nth-root to implement algorithmic tricks • Multiple pseudofermions are better • Mass preconditioning • Multiple timescales • Gluon, triple strange, light • Allows a larger overall step-size with good acceptance • Higher order integration schemes • RHMC algorithm 5-10 times faster • Binaries frozen since March 2006 AHM2008

  22. CPS: good and bad • CPS is written around target (QCDOC) hardware • Code base runs (correctly) on target hdw • Helps reduce risk when building own machine • Includes much requisite functionality • Adoption of CPS allowed UKQCD to focus on its strength • Very successful algorithmic development • Based on direct collaboration with RBC • Still need to do measurements • Invert fermion matrix (quark propagators) on gluon configurations • Do measurements on different architectures AHM2008

  23. Chroma/qdp++ • Open source c++ code base • Used by many different groups world-wide • Multi-platform by design • Highly modular, layered design • QMP: Wrapper around message passing library e.g. MPI • QDP++: Builds lattice valued physics data objects and manipulation methods • Hides message passing layer • Allows “Under-the-hood” optimisations by expert developers • Includes IO • Chroma The physics library • Rich physics functionality • UKQCD has historical links with main developers AHM2008

  24. qdp++ :: plaquette example Lattice valued data objects Manipulation methods multi1d<LatticeColorMatrix> u(Nd) for(int mu=1; mu < Nd; ++mu){ for(int nu=0; nu < mu; ++nu){ LatticeColorMatrix tmp_0 = shift(u[nu],FORWARD,mu) * adj(shift(u[mu],FORWARD,nu)); LatticeColorMatrix tmp_1 = tmp_0 * adj(u[nu]); Double tmp = sum(real(trace(u[mu]*tmp_1))); w_plaq += tmp; } } multi1d<LatticeColorMatrix> u(Nd) for(int mu=1; mu < Nd; ++mu){ for(int nu=0; nu < mu; ++nu){ LatticeColorMatrix tmp_0 = shift(u[nu],FORWARD,mu) * adj(shift(u[mu],FORWARD,nu)); LatticeColorMatrix tmp_1 = tmp_0 * adj(u[nu]); Double tmp = sum(real(trace(u[mu]*tmp_1))); w_plaq += tmp; } } multi1d<LatticeColorMatrix> u(Nd) for(int mu=1; mu < Nd; ++mu){ for(int nu=0; nu < mu; ++nu){ LatticeColorMatrix tmp_0 = shift(u[nu],FORWARD,mu) * adj(shift(u[mu],FORWARD,nu)); LatticeColorMatrixtmp_1 = tmp_0 * adj(u[nu]); Double tmp = sum(real(trace(u[mu]*tmp_1))); w_plaq += tmp; } } AHM2008

  25. qdp++ :: Abstraction • Data objects are lattice valued • No site index • No explicit sum over index • Linear algebra is encoded • Code knows how to multiply 3x3 matrices together • This has two consequences • Expert HPC developer can modify implementation • Optimisation, parallelism, architecture features • Interface remains the same • Application developer (Physicist) writes code which looks like maths! AHM2008

  26. qdp :: Code like maths multi1d<LatticeColorMatrix> u(Nd) for(int mu=1; mu < Nd; ++mu){ for(int nu=0; nu < mu; ++nu){ LatticeColorMatrix tmp_0 = shift(u[nu],FORWARD,mu) * adj(shift(u[mu],FORWARD,nu)); LatticeColorMatrix tmp_1 = tmp_0 * adj(u[nu]); Double tmp = sum(real(trace(u[mu]*tmp_1))); w_plaq += tmp; } } AHM2008

  27. qdp :: Code like maths multi1d<LatticeColorMatrix> u(Nd) for(int mu=1; mu < Nd; ++mu){ for(int nu=0; nu < mu; ++nu){ LatticeColorMatrix tmp_0 = shift(u[nu],FORWARD,mu) * adj(shift(u[mu],FORWARD,nu)); LatticeColorMatrix tmp_1 = tmp_0 * adj(u[nu]); Double tmp = sum(real(trace(u[mu]*tmp_1))); w_plaq += tmp; } } AHM2008

  28. qdp :: Code like maths multi1d<LatticeColorMatrix> u(Nd) for(int mu=1; mu < Nd; ++mu){ for(int nu=0; nu < mu; ++nu){ LatticeColorMatrix tmp_0 = shift(u[nu],FORWARD,mu) * adj(shift(u[mu],FORWARD,nu)); LatticeColorMatrix tmp_1 = tmp_0 * adj(u[nu]); Double tmp = sum(real(trace(u[mu]*tmp_1))); w_plaq += tmp; } } AHM2008

  29. Chroma: potential Downside • At time of QCDOC development didn’t have RHMC functionality • Heavy use of c++ templates can defeat some compilers • Stuck with Gnu compilers • Code is very advanced c++. Not easy for beginners • Main program driven by XML input files • All objects created on the fly • Requires a lot of functions to be registered • QCDOC has small memory (especially .text) • Chroma fails to compile on QCDOC • Runs out of .text segment • Physics librarycompiles OK AHM2008

  30. UKhadron • Old-style main program • Calls qdp++ and chroma library • Harness the power of qdp++ • Focused on UKQCD physics requirements • Most of measurement code for DWF project • Iterative solvers • Pros • Runs on QCDOC and everywhere • Control over code - small group of developers • Can build integrated analysis code on top of qdp++ • Cons • Compiling can be a headache! • UKhadron requires specific versions of qdp++/choma • Which require specific versions of Gnu compiles and libxml2 AHM2008

  31. BAGEL • Assembler generator written by Peter Boyle • http://www.ph.ed.ac.uk/~paboyle/bagel/Bagel.html • Composed in two parts • library to which one can programme a generic RISC assembler kernel • set of programmes that use the library to produce key QCD and linear algebra operations • generator is retargetable, key targets are ppc440, bgl, andpowerIII. • Allows kernels to run at up to 50% of peak on target arch AHM2008

  32. Modular Build • Both a blessing and a curse • Allows for modular, independent code development • Plug in performance code • Highly portable performance • Module version and compiler version dependence can be a problem QMP Libxml2 Bagel lib Bagel apps (bagel qdp, bagel wilson dslash) qdp++ Chroma UKhadron Can plug in other kernels eg SSE wilsonDslash AHM2008

  33. Future • Fastest machines are now BlueGene/P • Multicore • Cray XT4/BlackWidow • Multicore/vector machine • Multi-threading in qdp++: Mixed mode code • Shared memory intra-node • Message passing inter-node • PGAS languages? • UPC/CoArrayFORTRAN/Chapel/FORTRESS/X10? • Hardware designed for QCD (BlueGene/Q) • Performance kernel libraries AHM2008

  34. Physics • CPS on QCDOC Gluon cfgs • UKhadron on QCDOC +BlueGene/L + HECToR correlation functions • Implemented “twisted BC” in UKhadron  new calculation LQCD data (TW BC) Exp data • World’s best calculation of charge radius of pion • Can determine CKM matrix elements • Tests standard model at LHC • P.A. Boyle et alJHEP07(2008)112 AHM2008

  35. Conclusions • QCD very complex problem • Software is performance critical • Very complicated • UKQCD operates in a complex and changing • Collaborative environment • internally and externally • Hardware regime • Complex and evolving strategy • Allows maximum flexiblity • Gets the science done! AHM2008

More Related