1 / 16

Storage @ L’POOL

Storage @ L’POOL. HEPSYSMAN, July 09 Steve Jones sjones@hep.ph.liv.ac.uk. Procurement. Structure of the SE. Dual quad-core, 2.7 gHz, 64b, SL4.4, 10 g RAM glite-SE_DPM_mysql-3.1.13-0 Physical disk: 4 x Seagate 250GB, sda..d S/W Raid 1 md0,sda1,sdd1,spare, 15 GB, /

manju
Download Presentation

Storage @ L’POOL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Storage @ L’POOL HEPSYSMAN, July 09 Steve Jones sjones@hep.ph.liv.ac.uk

  2. Procurement

  3. Structure of the SE • Dual quad-core, 2.7 gHz, 64b, SL4.4, 10 g RAM • glite-SE_DPM_mysql-3.1.13-0 • Physical disk: 4 x Seagate 250GB, sda..d • S/W Raid 1 • md0,sda1,sdd1,spare, 15 GB, / • md1,sda2,sdd2,spare, 8 GB, swap • md2,sda3,sdd3,spare, 30 GB, /var • md3,sda5,sdd5,spare, 30 GB, /opt • S/W Raid 10 (Access and reliability for mySQL and /gridstore, which has storage for ops VO, SAMS tests) • md4, sda6, sdb6,sdc6,sdd6, 200 GB, /var/lib/mysql • md5,sda7,sdb7,sdc7,sdd7,113 GB, /gridstore

  4. Structure of the pool nodes • Dual quad-core, 2.3 gHz, 64b, SL4.4, 16 g RAM • glite-SE_DPM_disk-3.1.12-0 • Total of 8 systems • S/W Raid 1 (local for OS) • Physical disks: 2 x Seagate 250GB, hda, hdc • md0,hda1,hdc1,spare, 15 GB, / • md1,hda5,hdc5,spare, 4 GB, swap • md2,hda2,hdc2,spare, 30 GB, /opt • md3,hda3,hdc3,spare, 30 GB, /var • md4,hda6,hdc6,spare, 163 GB, /data

  5. Structure of the pool nodes • H/W Raid 6 (typical) • Physical disk: 24 x 1T SATA II drives, configured as 1 x 22 TB disk with 4 partitions. • sda1, 5.3TB • sda2, 5.3TB • sda3, 5.3TB • sda4, 5.3TB

  6. Structure of the pool nodes • Raid controllers, ~ 150 T • 3ware 9650SE-16ML • 3ware 9650SE-24M8 • Areca 1280

  7. The network • 1 gbps NICs, Intel • 4 per system. Two on MB + 2 extra • Each system has 2 network presences • One to HEP network, routable to Internet (138.253…), using 1 NIC. • Another for internal traffic (192.168.178…); busy, so 3 NICs, bonded (1 MAC) to Force10 switch (LACP). Useful when 100s of connections (can’t break a connection across two NICs). Isolated, local grid, allows jumbo frames).

  8. Each system has software firewall, to stop unwanted traffic (no ssh!) • Allows incoming rfio, gridftp, srm (head node only)

  9. STEP09 • CPU: ~ 20% • RAM: only occasionally in swap • NW: Close to saturation on pool nodes • S/W: • Fixes to fair shares • Update to Atlas Athena S/W (14.5.0)

  10. STEP09 Problems Rfio buffer sizes: We started with 128 MB buffers to load plenty of data (rfio). Many connections to pool nodes exhausted RAM. This slowed things down. Due to the lack of cache space, the disks got hammered. Fix: reduce buffer size to 64 MB. • Note: big buffers help CPU efficiency, but strategy fails if forced to use swap when RAM exhausted, or when bandwidth is saturated. It depends on the number of jobs/connections.

  11. STEP09 Problems Big data files:

  12. STEP09 Problems Small data files:

  13. STEP09 • We are ready for real data, we hope, as long as things remain as they are… • Improvements will continue, e.g. four more pool nodes of similar spec.

  14. Storage Allocation atlasPool, CAPACITY: 109.98TB (uses space tokens) Space Tokens: ATLASDATADISK, ATLASGROUPDISK, ATLASLIVERPOOLDISK, ATLASLOCALGROUPDISK, ATLASMCDISK, ATLASPRODDISK, ATLASSCRATCHDISK, ATLASUSERDISK Authorized FQANs: atlas, atlas/Role=lcgadmin, atlas/Role=production, atlas/lcg1, atlas/uk, atlas/uk/Role=NULL lhcbPool, CAPACITY: 21.99TB , USED: 2.37MB Authorized FQANs: lhcb, lhcb/Role=lcgadmin, lhcb/Role=production opsPool, CAPACITY: 116.28GB USED: 11.27GB Authorized FQANs: dteam, ops, ops/Role=lcgadmin t2kPool, CAPACITY: 22.03TB USED: 12.00TB Authorized FQANs: t2k, t2k/Role=lcgadmin, t2k/Role=production

  15. Our questions • Atlas data access is (apparently) random, which enormously reduces CPU efficiency by hammering the caching. Must this be so? Could it be sorted into some sequence?

  16. Thanks • Thanks to Rob and John at L’pool. • I hope it helps.

More Related