1 / 34

Using ATLAS@home to backfill grid sites

Using ATLAS@home to backfill grid sites. Wenjing Wu 1 , David Cameron 2 Computer Center, IHEP, China University of Oslo, Norway 2018-04-12. Outline . ATLAS@home ATLAS@home Backfilling clusters: BEIJING Tier2 site TRIUMF Tier1 site Integration of resource contributions

enlow
Download Presentation

Using ATLAS@home to backfill grid sites

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using ATLAS@home to backfill grid sites Wenjing Wu1, David Cameron2 Computer Center, IHEP, China University of Oslo, Norway 2018-04-12

  2. Outline • ATLAS@home • ATLAS@home Backfilling clusters: • BEIJING Tier2 site • TRIUMF Tier1 site • Integration of resource contributions • How to join as a site • Summary

  3. ATLAS@home ARC Control Tower Simulation Tasks PanDA queue: BOINC_MCORE • Started as volunteer computing project • Currently the major resources are from cluster nodes ARC CE BOINC Client Personal Computers BOINC Server ATLAS@home BOINC client Clusters

  4. Current scale (1) CPU time of good jobs of All ATLAS sitesin the past 10 days Avg. BOINC core power: 11HS06 BOINC: 3.52% CPU days: 8950 per day • Ranked as the 6th site in terms of good CPU time • Equivalent to a site with 140KHS06, given the avg. CPU util. of ATLAS site is less than 0.7

  5. ATLAS@home as a simulation site BOINC: 8.83% Avg. 4.1M events per day • Runs only simulation jobs, CPU intensive • Ranked as the 3rd site for simulation jobs • Over 50% of the ATLAS computing resources are used by simulation

  6. Backfilling Grid sites • Tier 2 site utilization rate • At BEIJING Tier2 site • At TRIUMF Tier1 site

  7. Lightweight model for ATLAS@home ARC CE ACT PanDA Volunteer Host (Windows/Mac) VirtualBox VM BOINC Server ATLAS app Cloud Nodes /PC/Linux servers Volunteer Host ( Linux) Singularity Grid sites Run directly Volunteer Host ( SLC/Centos 6)

  8. Tier2 is not always fully loaded • Site downtime • Conservative scheduling around downtime (draining jobs/ ramp up jobs slowly) • Not enough pilots at site (occasionally) • Very bad utilization for sites without dynamical partitioning when there is not enough simulation jobs (multi core jobs). • Avg. Wall Util. 85%, CPU Util. 70% • Wall_util=sum(wall_time)/cpu_cores • CPU_util=sum(cpu_time)/cpu_cores CPU Eff. For production jobs are between 12% to 96% Slides from Friedrich et al. CPU Eff. for user jobs are between 1% to 81%

  9. 100% wall != 100% CPU utilization due to job CPU Eff. Job 1: 12 Wall hour Job 1: 12 Wall hour Job 3: 12Wall hour 8 CPU hour 8 CPU hour 4 CPU hour • Job 2: 12 Wall hour • Job 2: 12 Wall hour • Job 2: Wall hour 8 CPU hour 8 CPU hour One work node With job 1-2, 100% wall utilization, assume job CPU Eff. 75%,then 25% CPU is wasted With job 1-4, 200% wall utilization, 100% CPU utilization, job eff 75% and 25% 4 CPU hour

  10. Put more jobs on work nodes • Run 2 jobs on each core • 1 grid job with normal priority (pri=20), 1 BOINC job with the lowest priority (pri=39) • Linux uses “non preemptive” scheduling for CPU cycles, which means high priority jobs occupies CPU until it releases the CPU voluntarily. • BOINC only gets CPU cycles when grid jobs do not use it. ATLAS Grid job ATLAS@home job

  11. Backfilling at BEIJING site • ATLAS Tier2 site (468 cores, managed by PBS) • IHEP local cluster: mixed jobs (ATLAS, CMS, BESIII, BelleII, JUNO etc.), managed by HTCondor

  12. BEIJING Tier2 site (in 100 days) • Grid jobs Walltime Util. is 87.8%, Grid CPU Util. is 65.6%, BOINC exploit an extra 23% of CPU time, node CPU Util. reaches 89% • More details from here: The overall CPU Util. is 98.44% on this node in 24 hours BEIJING_BOINC

  13. In Long term Weekly Consumed CPU days (2017.9-2018.2) 468 Cores CPU time by BOINC jobs Some downturns are caused by the BOINC server failure, hence not having enough jobs for the Tier 2 site to run

  14. At TRIUMF Tier 1 • Started about 2 weeks ago • The site has 4844 cores in total, dedicated for ATLAS • They started ATLAS@home on all their work nodes

  15. Extra CPU time! • Red Bins represent CPU time which would be otherwise wasted • Translated into 1300 CPU days per day • Counts 31.79% CPU time of the site delivers Check here for your site

  16. Improvement of CPU Util. Before: • CPU_Util.=cputime_alljobs_in_a_day/number_of_cores • Before adding ATLAS@home, CPU Util. is 0.68 • After adding ATLAS@home, CPU Util. is 0.90, improves by 0.22 • No impact to Wall Util. of grid jobs (0.91 vs. 0.93) • It might affect the CPU Eff. Of grid jobs, we are observing it in a long run After:

  17. DBRSC(Dynamical BOINC ReSource Configuration) • ATLAS@home jobs : 2-12 cores per job, avg. 2-4 CPU hours per job • Short jobs can lose up to 27% CPU Eff. with big core per job. • Trade-off between Memory usage and CPU Eff.

  18. DBRSC: Dynamical BOINC ReSource Configuration • Takes control of BOINC resource configuration, hide the BOINC details from site admins • Do not need to know the hardware configuration and resource usage of the non-BOINC jobs on the computer, DBRSC decides how much CPU/Memory BOINC can use according to the historical resource usage of non-BOINC jobs dynamically. • Configure core_per_job dynamically before starting a job according to available resource to BOINC, to increase job CPU Eff. • Making sure BOINC jobs do not impact non-BOINC jobs. Kill BOINC automatically if abnormal system load appear, restart BOINC automatically when system load returns normal.

  19. Effect of DBRSC On the same set of nodes (ATLAS single core analysis nodes), CPU Util. is increased by 15% (from 74.4 % to 89.4%) while the Grid workload is the same (48%), 15% increase of BOINC CPU Node CPU Util. is 74.4% before DBRSC Node CPU Util. is 89.4% After DBRSC

  20. Resource Contribution Integration • Goal: • Sites can use BOINC to manage Tier 3 clusters, or do backfilling on clusters. • Recognize the site’s BOINC resource contribution • Sites’ BOINC contribution can be integrated into their Grid sites. • How • Site can create an ATLAS@home account, and run ATLAS@home on any computers under this account • The account name should be the same as their AGIS site name. • TRIUMF-LCG2, BEIJING-LCG2

  21. Demo visualization from Kibana • BOINC nodes from BEIJING_LCG2 belongs to a fake PanDA queue BEIJING_BOINC • Daily average: BEIJING_LCG2 site provides 640 CPU days (1310 Wall days ), and 365 CPU days (924 Wall days) is from BOINC. We also run BOINC on our Tier3 nodes

  22. In sites resource contribution ranking • Sites’ BOINC contribution can also be integrated into the sites in the resource contribution ranking • TRIUMF contributes 1.43% Without adding BOINC, 1.92% with adding BOINC to the whole ATLAS computing resources These demo visualizations/dashboards are created here in Kibana. Dario is pushing to include it in the official monitoring.

  23. Quick start to run BOINC at site • The BOINC client is installed in cvmfs • /cvmfs/atlas.cern.ch/repo/sw/BOINC/BOINC • You only need to configure BOINC on your work node • cd /cvmfs/atlas.cern.ch/repo/sw/BOINC/BOINC • source setup.sh • sh config_boinc.sh [–uid your_boinc_authenticator] [–proxy hostname_proxy_server] • We have a default uid, and proxy is for work nodes behind firewall • Now, BOINC is running, and this includes the DBRSC! • Optimize the resource allocation/configuration dynamically • Maintain the sanity of BOINC (kill or start according to system load) • The command Boinc_agent is all you need BOINC can be run as any user! Non SLC/CC 6 nodes need singularity

  24. Detailed document in Twiki • For sites who want to join, we provide all the instruction documents here: Steps to follow if you want to install and configure your own BOINC, otherwise, the quick start recipe works!

  25. You can also install the BOINC client with an all in one script to deploy on cluster: installing BOINC, start running ATLAS jobs, dynamically configure BOINC resources.

  26. Summary • ATLAS@home is becoming a big resource contributor to ATLAS, and the resource is stable and reliable • Backfilling on the BEIJING ATLAS grid site exploit an extra of 28% CPU(6 months), on regular cluster is 46%. • Sites are encouraged to use ATLAS@home to harness their non official ATLAS computing resources and Backfilling running it on the clusters. • Dynamical BOINC configuration makes sure: • Efficiently exploit the available resource • Not to affect the Non BOINC jobs • Hide the BOINC details to site admins.

  27. Acknowledgements • Andrej Filipcic (Jozef Stefan Institute, Slovenia) • Simone Campana (CERN) • Nils Hoimyr (CERN IT, BOINC Project) • Ilija Vukotic (University of Chicago) • Frank Berghaus (University of Victoria) • Douglas Gingrich (University of Alberta) • Paul Nilsson (BNL) • Rod Walker ( LMU-München)

  28. Thanks!

  29. Backup slides

  30. All BEIJING Nodes • About 1300 cores running BOINC, extra 45.55% CPU time is exploited by BOINC, node Avg. CPU utilization reaches 88.6%. (98.44 % in the best case in 24 hours) • Translate to 595 CPU days dailyin ATLAS@home • Implemented DBRSC (generic to BOINC) CPU: 88.4% Non-BOINC , 10% BOINC 1.25K BOINC Processes CPU: 43% Non-BOINC , 45.6% BOINC

  31. BOINC scalability • The current ATLAS only use 2% of LHC@home resource. • LHC@home is just one BOINC server, handles 0.7M jobs per day in peak time, ATLAS only processes 10K jobs per day. ATLAS jobs

  32. Impact on Tier2? • Stability • It does not affect the production job failure rate (1-2%) • It does not affect the throughput of production jobs • with running BOINC, ATLAS grid jobs walltime utilization is around 90% • Work nodes have reasonable load/swap usage • Efficiency • For production jobs , it does not make it less efficient • CPU efficient is less than 1% difference or NONE?! • For BOINC jobs, it does not make it less efficient • CPU time per event is 0.02% difference between dedicated nodes and nodes mixed with production jobs. • Manpower to maintain • After the right configuration, no manual intervention is needed

  33. Production jobs 1. Production jobs in a month: (with BOINC jobs) 1-2% failure rate 2. In recent 3 weeks, production jobs uses 90% of the walltime

  34. Job Efficiency compare

More Related