filesytems and file access n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Filesytems and file access PowerPoint Presentation
Download Presentation
Filesytems and file access

Loading in 2 Seconds...

play fullscreen
1 / 22

Filesytems and file access - PowerPoint PPT Presentation


  • 70 Views
  • Uploaded on

Filesytems and file access. Wahid Bhimji University of Edinburgh, Sam Skipsey , Chris Walker …. This session:. 2 Themes: What are the demands made on storage by LHC data analysis – how might they change What solutions ( filesystems or other) help us to meet such demands.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Filesytems and file access' - jerom


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
filesytems and file access

Filesytems and file access

Wahid Bhimji

University of Edinburgh,

Sam Skipsey, Chris Walker ….

Wahid Bhimji – Files access

this session
This session:

2 Themes:

  • What are the demands made on storage by LHC data analysis – how might they change
  • What solutions (filesystems or other) help us to meet such demands

Wahid Bhimji – Files access

demands data access
Demands - Data access
  • LHC data analysis at Tier2s involves large files read from servers over the network.
  • Significant load and bottlenecks on storage. Tuning is affected by :
      • File access pattern
      • File size (e.g. whether can be cached in RAM)
      • Access method (direct local access, copy to WN etc.)
      • And more…
  • Problems can be “fixed” by the VO or site.

Wahid Bhimji – Files access

examples rfio access
Examples – rfio access

A reminder of STEP last year (Sam/ Glasgow)

  • “Rfio” access
  • Large data files.
  • “random” file access
  • Large rfio buffers

–> Disk server Memory exhausted, network saturated

Solution Employed:

  • Smaller RFIO buffers and copy to WN

But …..

Wahid Bhimji – Files access

copy to wn moves the problem
copy to WN – moves the problem ..
  • More recent tests using FILESTAGER
  • Disk seek count hits max for drive

Solution:

  • SSDs (see Sams talk)?
  • More WN disks?

Or …:

Smaller files/ Sequential access ?....

Wahid Bhimji – Files access

heart of the matter root i o
Heart of the matter - ROOT I/O
  • Fundamental to most LHC analysis…
  • Need not be the bad guy.

Root Data structure – “Tree”

has multiple “Entries” – (physics “events”)

and “Branches” (Objects – e.g. a Track).

Wahid Bhimji – Files access

problems
Problems
  • Order that elements of Tree written into file
  • Ordering by branches –compresses well - good for extracting a single branch
  • BUT: in experiments data, there are lots of branches, branches are uneven sized, buffers are fixed (and small) so
  • Branches for the same event are scattered in the file

Wahid Bhimji – Files access

aside on testing
Aside on testing
  • “ROOT Test” - Read through the Tree with Tree->GetEntry, observe pattern with TTreePerfStats
  • Athena / CMSSW - Real analysis jobs
  • Hammercloud tests - Multiple continuous athena jobs

Wahid Bhimji – Files access

random access root test on atlas aod file
“Random” access – ROOT test on Atlas AOD file
  • RFIO access

(128k buffer)

  • Scatter in read
  • Poor cpu Efficiency

(~ 15 % )

Wahid Bhimji – Files access

tuning filesystem can make a difference
Tuning / Filesystem can make a difference
  • GPFS: 3 x faster than rfio access (in this test for this file)
  • Buffering

Wahid Bhimji – Files access

cms similar picture
CMS – similar picture

Brian

Bockelman

https://twiki.cern.ch/twiki/bin/view/Sandbox/CmsIOWork

Wahid Bhimji – Files access

reordering helps
Reordering helps
  • Reordered ATLAS AOD Tree using CloneTree(-1,"SortBasketsByEntry");
  • Factor 5 quicker in reading through Entries (or rfio)

Wahid Bhimji – Files access

new atlas files
“new” ATLAS files
  • Recentlyboth CMS and ATLAS have been optimising no. of branches and writing baskets in event order.
  • Files are clearly reordered - CPU eff = ~90% (for this ROOT test)

Wahid Bhimji – Files access

hammercloud local access cpu eff event rate
HammercloudLocal Access CPU eff / event rate

Indicative results – many things can effect these – further testing needed

http://gangarobot.cern.ch/hc/1243/test/

http://gangarobot.cern.ch/hc/1246/test/

Wahid Bhimji – Files access

copy to wn with reordered files
Copy to WN with reordered files
  • HDD

< 120

Seeks/s

Much

Happier!

Wahid Bhimji – Files access

ttree caching
TTree Caching
  • Group into a buffer all blocks from used branches, sort in ascending order and merge so read sequentially.
  • Learn from the first few events what branches are of interest.
  • Claims to reduce “typically by a factor 10000 the number of transactions with the disk and in particular the network with servers like xrootdor dCache.” (vector reads)
  • Configuration still under investigation
    • Size of cache / training
    • Behavior with rfio – inc. possible segfaults.
    • Not yet used “by default” in ATLAS – config likely to be site specific

Wahid Bhimji – Files access

shows further improvements
Shows further improvements

e.g. Dcache- GuenterDuckeck

Wahid Bhimji – Files access

conclusions
Conclusions
  • I/O pattern of software can cost money for site
  • Much of this is closely coupled to storage choices and configuration
  • Close feedback from sites to VOscan help access method (and site tuning) to become (and stay) optimal
  • Hotch-Potch of evidence – could do with systematic study

Wahid Bhimji – Files access

memory tree each node is a branch in the tree
Memory <--> TreeEach Node is a branch in the Tree

Memory

T.GetEntry(6)

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

T.Fill()

18

T

Rene Brun - Sinaia

Wahid Bhimji – Files access

tr

root i o split cluster tree version
ROOT I/O -- Split/ClusterTree version

Tree entries

Streamer

Branches

Tree in memory

File

Wahid Bhimji – Files access

andrew lahiff ral cms
Andrew Lahiff – RAL - CMS

Reconstruction (4000 events)

Skimming (4000 events)

Wahid Bhimji – Files access