1 / 5

Computing Update for 12 GeV User Group Board of Directors Meeting

Chip Watson, Deputy CIO, discusses data challenges, farm growth plans, and workflow improvements for petabytes at the FY14-16 meetings. Challenges include expanding capacity, node sharing, and node lending. The focus is on testing downstream processes. Workflow evolution tools are outlined for efficient data management.

lavonn
Download Presentation

Computing Update for 12 GeV User Group Board of Directors Meeting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computing UpdateData Analysis (farm) for 12 GeVUser Group Board of Directors MeetingChip WatsonScientific Computing, Deputy CIO Outline Data challenges, farm capacity growth Plans for petabytes Workflow & related topics

  2. Quick Overview of Expansions FY14: Not much happening. Improve software & operations. FY15: First major 12 GeV farm upgrade (5K-6K cores) FY16: Major LQCD upgrade Second major 12 GeV farm upgrade (tbd) Add second tape library

  3. Data Challenges for 12 GeV Goal: 10% scale 24 months in advance 25% scale 18 months in advance 50% scale 12 months in advance 100% scale 6 months in advance Test everything downstream of data acquisition • transfer of data from hall to data center • near-live analysis (data buffer on disk) • push to tape • pull from tape + offline analysis

  4. Data Challenges for 12 GeV Farm / LQCD node sharing: move nodes Hall D: online at 5000 cores May 2015 10% done 25% Feb 2014, will loan 1K+ cores, so farm is at 2.2-2.5K, with Hall D using half, so simulating real competing load 50% late summer 2014, will loan 2K – 2½ K cores, and might allow ongoing use of 1000 cores until FY15 cluster comes online 100% January 2015, new FY15 farm nodes go online, support final data challenge

  5. Offline 2014 Evolution Workflow tools • define & track a “workflow”, consisting of many jobs, tasks, file I/O operations • auto-retry on failed jobs • way to query (or see online) how much progress the workflow has achieved • add / remove tasks from workflow as it is running Write through disk cache • never fills, overflows to tape • can be used by Globus Online WAN file transfers to write to Jlab tape library Stage-out unused work disks

More Related