1 / 56

“UCF”

“UCF”. Computing Capabilities at UNM HPC. Timothy L. Thomas UNM Dept of Physics and Astronomy. I have a 200K SU ( 150K LL CPU hour ) grant from the NRAC of the NSF/NCSA, with which UNM HPC (“AHPCC”) is affiliated. (Stolen from Andrew…). Peripheral Data Vs Simulation.

adia
Download Presentation

“UCF”

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. “UCF” Computing Capabilities at UNM HPC Timothy L. Thomas UNM Dept of Physics and Astronomy Timothy L. Thomas

  2. Timothy L. Thomas

  3. Timothy L. Thomas

  4. Timothy L. Thomas

  5. Timothy L. Thomas

  6. Timothy L. Thomas

  7. Timothy L. Thomas

  8. I have a 200K SU (150K LL CPU hour) grant from the NRAC of the NSF/NCSA, with which UNM HPC (“AHPCC”) is affiliated.

  9. (Stolen from Andrew…) Peripheral Data Vs Simulation Simulation: Muons From Central Hijing (QM02 Project07) Data: Centrality by Perp > 60

  10. (Stolen from Andrew…) Simulated Decay Muons • QM’02 Project07 PISA files (Central HIJING) • Closest cuts possible from PISA file to match data(PT parent >1 GeV/c, Theta Porig Parent 155-161) Investigating possibility of keeping only muon and parent hits for reconstruction. • 17100 total events distributed over Z=±10, ±20, ±38 More events available but only a factor for smallest error bar Zeff ~75 cm Not in fit "(IDPART==5 || IDPART==6) && IDPARENT >6 &&IDPARENT < 13 && PTHE_PRI >155 && PTHE_PRI < 161 && IPLANE == 1 && IARM == 2 && LASTGAP > 2002 && PTOT_PRI*sin(PTHE_PRI*acos(0)/90.) > 1."

  11. Now at UNM HPC: • PBS • Globus 2.2.x • Condor-G / Condor • (GDMP) • …all supported by HPC staff. • In Progress: • A new 1.2 TB RAID 5 disk server, to host: • AFS cache  PHENIX software • ARGO file catalog (PostgreSQL) • Local Objectivity mirror • Globus 2.2.x (GridFTP and more…)

  12. Pre-QM2002 experience • with globus-url-copy… • Easily saturated UNM • bandwidth limitations • (as they were at that time) • PKI infrastructure and • sophisticated error-handling • are a real bonus over bbftp. • (One bug, known at the time • is being / has been addressed.) • (at left: 10 streams used) KB/sec

  13. Multi-Jet cross section (theory) calculations, run using Condor(/PVM)… Three years of accumulated CPU time on desktop (MOU) machines at HPCERC and at the University of Wisconsin. Very CPU-intensive calculations… 6- and 9-dimensional Monte Carlo integrations: A typical job runs for a week and produces only about 100 KB of output histograms, such as those displayed here.

  14. LLDIMU.HPC.UNM.EDU Timothy L. Thomas

  15. Timothy L. Thomas

  16. Timothy L. Thomas

  17. Timothy L. Thomas

  18. Timothy L. Thomas

  19. RAID op system issues • Easy re-installation / update of the op sys • Grub or Lilo? (MBR or /boot?) • Machine has an IDE CDROM (but not a burner)!!! • Rescue CDs and/or floppies… • Independence of RAID array • (1.5 hours for RAID 5 verification step) • Should install ext3 on the RAID. • Partitioning of the system disk: • Independence of /home area • Independence of /usr/local area? • Jonathan says: Linux can’t do more than 2 GB swap partition • Jonathan says: / /usr/local/ /home/ (me: /home1/ /home2/ …?) • NFS issues… • Synchronize UID/GIDs between RAID server and LL. Timothy L. Thomas

  20. RAID op system issues • Compilers and glibc… Timothy L. Thomas

  21. RAID op system issues • File systems… • What quotas? • Ext3? (Quotas working OK?) • ReiserFS? (Need special kernel modules for this?) Timothy L. Thomas

  22. RAID op system issues • Support for the following apps: • Raid software • Globus… • PHENIX application software • Objectivity • gcc 2.95.3 • PostGress • Open AFS  Kerberos 4 Timothy L. Thomas

  23. RAID op system issues • Security issues… • IP#: fixed or DHCP? • What services to run or avoid? • NFS… • Tripwire or equiv… • Kerberos (for Open AFS)… • Globus… • ipchains firewall rules; /etc/services; /etc/xinetd config; etc… Timothy L. Thomas

  24. RAID op system issues • Application-level issues… • Which framework? Both? • Who maintains framework and how can this job be properly divided up among locals? • SHOULD THE RAID ARRAY BE PARTITONED, a la the PHENIX counting house buffer boxes’ /a and /b file systems? Timothy L. Thomas

  25. Resources Assume 90KByte/event and 0.1GByte/hour/CPU Filtered event can be analyzed, but not ALL PRDF event Many trigger has overlap.

  26. Rough calculation of real-data processing (I/O-intensive) capabilities: 10 M events, PRDF-to-{DST+x}, both mut & mutoo; assume 3 sec/event (*1.3 for LL), 200200 KB/event.  One pass: 7 days on 50 CPUs (25 boxes), using 56% of LL local network capacity.  My 200K “SU” (~150K LL CPU hours) allocation allows for 18 of these passes (4.2 months)  3 MB/sec Internet2 connection = 1.6 TB / 12 nights (MUIDN_1D1S&NTCN) (Presently) LL is most effective for CPU-intensive tasks: simulations can easily fill the 512 CPUs; e.g, QM02 Project 07. Caveats: “LLDIMU” is a front-end machine; LL worker node environment is different from CAS/RCS node ( P.Power…)

  27. Timothy L. Thomas

  28. Timothy L. Thomas

  29. On UNM Grid activities T.L.Thomas Timothy L. Thomas

  30. I have a 200K SU (150K LL CPU hour) grant from the NRAC of the NSF/NCSA, with which UNM HPC (“AHPCC”) is affiliated.

  31. . CPU time used: ~ 33,000 LosLobos hours . Number of files handled: > 2200 files . Data moved to BNL: > 0.5 TB (globus-url-copy) (NOTE: In 2001, did even more (~110,000 hours), as an as an exercise… see http://thomas.phys.unm.edu/tlt/phenix_simulations/ ) . Comments: [from late summer... but still relevant] . Global storage and I/O (disk, network, network) management a headache; too human intensive. --> Throwing more people at the problem (i.e., giving people accounts at more remote sites) is not a particularly efficient way to solve this problem. . File naming standardessential (esp. for data base issues.) . I have assembled a (still rough; not included here) standard request form for DETAILED information... --> This could be turned into an automatic interface... A PORTAL (to buzz) . PWG contacts need to assemble as detailed a plan as they can, but without the kinds of system details that are probably going to be changed anyway. (e.g., "chunk" size hints welcome but may be ignored.) . Use of varied facilities requires flexibility, including an "ATM" approach --> Simulation database needs to reflect this complexity.

  32. . Generator config / management needs to be somewhat more sophisticated. --> E.g., random seeds, "back-end" generation. . An big issue (that others may understand better): the relationship and interface between the simulation data base and other PHENIX data bases... . Multiple levels of logs actually helped bookkeeping! --> Perhaps 'pseudo-parallelism' is the way to go. . Emerging reality (one of the main motivations "Grid" technology): no one has enough computing when it's needed but everyone has too much when they don't need it, which is much of the time. More than enough computing to get the work done is out there; you don't need your own! BUT: they they are "out there", and this must be dealt with. ==> PHENIX can and should form its own IntraGrid.

  33. Reality Check #1: Perpetual computing person-power shortage; this pertains to both software production and data production, both real and M.C. Given that, M.C. is presently way too much work. Simple Vision:Transparently distributed processing should allow us to optimize our use of production computing person- power. Observed and projected massive increases in network bandwidth make this a not-so-crazy idea. Reality Check #2: What? Distributed real-data reco? Get Real! (...?) Fairly Simple Vision:OK, OK: Implement Simple Vision for M.C. first, see how that goes. If one can process M.C., then one is perhaps 75% of the way to processing real data. (Objy write-back problem is one serious catch.)

  34. (The following slides are from a presentation that I was invited to give to the UNM-led multi-institutional “Internet 2 Day” this past March…) Timothy L. Thomas

  35. Internet 2 and the Grid The Future of Computing for Big Science at UNM Timothy L. Thomas UNM Dept of Physics and Astronomy Timothy L. Thomas

  36. Grokking The Grid Grok v. To perceive a subject so deeply that one no longer knows it, but rather understands it on a fundamental level. Coined by Robert Heinlein in his 1961 novel, Stranger in a Strange Land. (Quotes from a colleague of mine…) Feb 2002:“This grid stuff is garbage.” Dec 2002:“Hey, these grid visionaries are serious!” Timothy L. Thomas

  37. So what is a “Grid”? Timothy L. Thomas

  38. Ensemble of distributed resources acting together to solve a problem: ”The Grid is about collaboration, about people working together.” • Linking people, computing resources, and sensors / instruments • Idea is decades old, but enabling technologies are recent. • Capacity distributed throughout an infrastructure • Aspects of Grid computing: • Pervasive • Consistent • Dependable • Inexpensive Timothy L. Thomas

  39. Virtual Organizations (VOs) • Security implications • Ian Foster’s Three Requirements: • VOs that span multiple administrative domains • Participant services based on open standards • Delivery of serious Quality of Service Timothy L. Thomas

  40. High Energy Physics Grids • GriPhyN (NSF) • CS research focusing on virtual data, request planning • Virtual Data Toolkit:Delivery vehicle for GriPhyN products • iVDGL: International Virtual Data Grid Laboratory (NSF) • A testbed for large-scale deployment and validation • Particle Physics Data Grid (DOE) • Grid-enabling six High-Energy/Nuclear Physics experiments • EU Data Grid (EDG): Applications areas… • Particle physics • Earth and planetary sciences: "Earth Observation“ • “Biology” • GLUE: Grid Laboratory Uniform Environment • Link from US grids to EDG grids Timothy L. Thomas

  41. <<< Grid Hype >>> (“Grids: Grease or Glue?”) Timothy L. Thomas

  42. Natural Grid Applications • High-energy elementary particle and Nuclear Physics (HENP) • Distributed image processing • Astronomy… • Biological/biomedical research; e.g., pathology… • Earth and Planetary Sciences • Military applications; e.g., space surveillance • Engineering simulations  NEES Grid • Distributed event simulations • Military applications; e.g., SF Express • Medicine: distributed, immersive patient simulations  Project Touch • Biology: complete cell simulations… Timothy L. Thomas

  43. Processing requirements Two examples • Example 1: High-energy Nuclear Physics • 10’s of petabytes of data per year • 10’s of teraflops of distributed CPU power • Comparable to today’s largest supercomputers… Timothy L. Thomas

  44. flow of data EpoDB GAIA Biological Databases: Complex interdependencies • Domino-effect in data publishing • Efficiently keep many versions GERD TRRD BEAD EpoDB BEAD Swissprot Swissprot GAIA EMBL GenBank Transfac Transfac DDBJ (Yong Zhao, University of Chicago)

More Related