Status of the BaBar Databases - PowerPoint PPT Presentation

status of the babar databases n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Status of the BaBar Databases PowerPoint Presentation
Download Presentation
Status of the BaBar Databases

play fullscreen
1 / 29
Status of the BaBar Databases
188 Views
Download Presentation
zia-rodriguez
Download Presentation

Status of the BaBar Databases

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Status of the BaBar Databases Jacek Becla BaBar Database Group

  2. BaBar Is in Production • Run 1: May 1999 – Oct 2000 • ~24.2 fb-1 (~1.3 per month) • Run 2: Feb 2001 – July 2002 • up to 12.6 fb-1 now (~2.5 per month) • Expected ~100 fb-1 by July 2002 • already well over designed luminosity

  3. Prognosis

  4. Changes • 4 -> 21 streams • >5 times more files, locks • no data duplication (streams not self-contained) • Smaller files • 2 -> 0.5, 10 -> 2 [GB] • Using Objy 6.1, read only dbs • Clustering hint server and cond OID server • Migrating production to Linux (now) • Introducing multi-fds (now) • Cannot afford a large test-bed anymore

  5. OPR • In general keeps up with data • ~150 pb-1 per day • faster than at the end of Run 1 • in spite of 5x load • will have to deal with 300 pb-1 soon

  6. Current OPR Configuration • Hardware • 6 4-CPU data servers, lock server, jnl server, catalog server, clustering hint server + conditions OID server • 220 clients • Software • Objy 6.1, Solaris 7 • about to migrate to Linux

  7. OPR – Short Term Future • Use multi-fds • 2 event store fds, 1 conditions • 6 + 6 data servers • new federation approx. every week • Migrate clients to Linux • 2.2 faster CPU, more memory • Use faster machine for lock servers • now: Sun Netra T1, 440 MHz • planned: Sun Blade 1000, 750 MHz UltraSPARC-3 • Discussions about storing all digis in objy, and reprocessing from Objy, not xtc

  8. REPRO • Hardware configuration similar to OPR • Occasionally up to 3 repro farms • over 300 pb-1 on a good day • 150+150+200 nodes • condition merging nightmare

  9. REPRO – Near Future • Use multi-fds • 2 event store fds, 1 conditions • 5 + 5 data servers • new federation ~ every other week • same slow lock servers • Move to Linux • Run in Italy. Timescale ~mid 2002

  10. Robustness • Db creation (weak point) removed • precreation in background by CHS, automatic recovery, new C++ api in 6.0 • AMS crash • ¾ of the farm continues, unless it is a “default” AMS(used by CHS) • CHS – new central point of failure • entirely in our hands, very stabile so far • One event store fd down (e.g. lock server crash) • the second should finish processing current run • Cleanup server – worked on

  11. Analysis • 200 CPUs (~Sun Netra T1 like) • 17 servers, 24 TB disk cache • On demand staging turned off • Read only dbs • starting to see effect now • Disk space – always a problem • micro – 5.4 KB/event (aod, col, tag, evt, evshdr) • mini – 4.7 KB/event (esd)

  12. Analysis – cont… • Veritas File System reconfiguration • direct I/O instead of buffered I/O • more than doubles effective data rate • Lock server memory leak • grows up to 600 MB in a week • switching every week • Kanga (ROOT based) will become deprecated • recent computing model: enhance Objy, deprecate kanga (freeze by Mid 2002, produce files till late 2002)

  13. AMS • Known (but not fixed) problem • file used immediately after being closed • crashes AMS (in 6.1 kills the client) • Ported to Linux • no performance figures yet • New feature - compression • Redesigning front end part • got ok from Objy

  14. A Word on Conditions • Using OID server to find time interval • only in REPRO so far, about to put in OPR • Staircase problem • incorrect design • purging every 2 weeks, ~15 min per rolling calibration (35 in total), run in parallel • Finalize problem • based on genealogy object, (all objects named), result of iteration in unpredicted order. Just slow • Condition merging problem

  15. Conditions…cont • Index problem • occasionally index inconsistent (does not return all objects in given range). Solution – rebuild. Happens ~once every 2 months. Not reported yet. • Index scaling • range query (the way we use it) does not scale • response time linear (100 K entries -> 0.5 sec) • Will extend OID server • now read only access • Will redesign & re-implement conditions • and address all the problems, timescale: end of 01

  16. Data Distribution • Micro-level data mirrored @ in2p3 • Run2 – mirror raw as well • Current tools do not scale with increased data volume • a lot of manual work • Will try using data grid based tools soon

  17. Operations • 2 DBAs +3rd coming soon • Many manual tasks slowly being automated

  18. Some Numbers • Total size of data – 300+ TB • # files – 128K • # users in analysis ~220 • 10 active production federations • this includes 5 analysis fds • Cond dbs – 12 GB

  19. TuningPerformanceScalability

  20. 4 streams: 100 nodes: ~ 60 Hz200 nodes: ~115 Hz 160 nodes run, 20 streams, with duplication 8 Hz 420 Streams Was Non-trivial

  21. Clustering Hint Server • CORBA based, multithreaded • Precreates in background dbs and conts, distributes oid to clients • Many other features: • containers reused • full integration with HPSS (precreated files pinned in cache, full dbs immediately migrated) • file disparsification • file transfer to tape: 1MB -> 15-25MB now • db creation locally, pre-sizing • no container extensions on the client side • round robin load balancing • automatic recovery, and so on

  22. Others • commitAndHold • significant reduction in lock traffic • Initial transaction for condition • one instead of 50 transactions • Cache authorization • rather then check on every event • Tune # client file descriptor limit • Hit 8K limit on AMS site. Reduced client fd limit: 196 -> 32. AMS response improved, AMS CPU usage decreased • Increase trans granularity

  23. Bottlenecks • Lock server • 1st signs of saturation: with ~ 200 nodes • use faster CPU • use Objy 7 (33% lock traffic reduction) • scheduled for October 2001 • more event store fds per farm • CPU on data servers • buy more – expensive • improve AMS, reduce event size

  24. Use Faster CPU…

  25. Miscellaneous • 64 K pages? • unfortunately not working with multi-fds • Maybe precreate/purge dbs only in between runs? • David is stepping down as a head of the BaBar DB group

  26. Future Looks Bright • Lock server bottleneck • multi-fds – can always add one more event store fd • Objy 7 will feature faster lock server • CPUs are getting faster • Data server CPU saturation • AMS redesign should help • size of event (rec) being reduced now by ~10%, looking for more • can always buy more servers

  27. Summary • No serious problems • conditions need to be redesigned • Likely OPR will keep up • Working in the BaBar DB group is fun!