1 / 15

CDF Taking Stock

CDF Taking Stock. By Anil Kumar CD/CSS/DSG Jul 26, 2006. Current Infrastructure. Current Infrastructure. CDF Capacity All Applications. CDF Offline DB Growth* 48GB/year * Slow Control is not in Offline CDF Online DB Growth 76G/year vs 50GB last year. CDF Online Applications.

tracey
Download Presentation

CDF Taking Stock

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CDF Taking Stock By Anil Kumar CD/CSS/DSG Jul 26, 2006 CDF Taking Stock 2005-2006

  2. Current Infrastructure CDF Taking Stock 2005-2006

  3. Current Infrastructure CDF Taking Stock 2005-2006

  4. CDF Capacity All Applications • CDF Offline DB Growth* 48GB/year * Slow Control is not in Offline • CDF Online DB Growth 76G/year vs 50GB last year. CDF Taking Stock 2005-2006

  5. CDF Online Applications CDF Taking Stock 2005-2006

  6. CDF Offline Applications CDF Taking Stock 2005-2006

  7. Uptimes • Cdfonprd 100% • Cdfofprd 99.9589% Vs 99.4356% last year. 225 minutes unscheduled Down Time since June 2005 • Cdf Replica 100% CDF Taking Stock 2005-2006

  8. Accomplishments • Upgraded CDF databases to 10gR2 and streams Deployment on 10gR2 • Upgrade OEM to 10g • Quarterly Database Security/OS patches Up-to-date. • Set-up the streams integration environment. Strmrep v2_2 in production. Case Study of Streams Replication at Fermi was presented in Oracle Open World 2005 This is major milestone in implementation of streams. • Rebuilt of cdfonline dev/int databases on rebuilt b0dau36 machines. OS upgrade to 64 bit 2.9 and Db upgrade to 10gR2. • Rewrite of dbatools/toolman for enhanced features of monitoring and 10g support. • Db Security Enhancements. Restricting access to Dictionary. Restricted Usage of Database Links. Password complexiety,locking the obsolete accounts and password complexity. • Deployment of SAM Request System Schema v6_0, v6_1 and V6_3 in development. • Standardized the rman_dcache product. Archives being sent to Dcache. Needs some more fine tuning. • Moving  backup to a standardized backup recovery method using a san and enstore.  • Full Backups of cdfofpr2 going in production this week. Ray P. gave 3TB SAN. This will facilitate the full recovery of cdfofpr2. CDF Taking Stock 2005-2006

  9. Back-up • CDF ONLINE DATABASES cdfonprd - Daily, 5 days of archives, Two Backup copies always on DISK - Allocated 939GB, Used 722 GB ( 2 Copies) , Backup time: 1 Hr 30 Mins - CDF on-line Backup to DCache/Enstore: Daily Archives to Dcache: every 30 mins. cdfondev Daily, 14 days archives, one always on DISK Used : 52GB , Backup time : 1 Hrs 15Min cdfonint Daily, 7 days archives, one always on Disk Used 180 GB Backup Time: 4 Hrs 15 Min Note : Total Backup Space allocated for dev and int is 337G • CDF OFFLINE DATABASES cdfofprd Full Daily, 8 days of archive. Two on NAS. Export of DFC + SAM. Partial backup on Local will be discontinued soon. This area may be used for archives. Allocated 3T, Used 388 GB Backup time -> 2 Hrs 26 Mins Cdfstrm1 being replica of on-line and DFC. No backup ->RMAN/ Tape. cdfofdev– Weekly cdfofdev, 7 days of archives One always on DISK Used : 17GB Backup Time : 2 Hrs 40 Mins cdfofint 2 times/week for cdfofint only for SAM+DFC+BOOKS, 8 days of archives Used: 33GB Backup Time : 2:36Mins Note : Total Backup space allocated for dev and int is 67GB CDF Taking Stock 2005-2006

  10. Oracle Backup for cdfonprd toDCache/Enstore • RMAN to DCache/Enstore is working fine, but needs fine tuning to fit our(dsg) standard. • Working reliably. Fully automated for dailys. • Using daily, weekly and monthly PNFS directory structure. • Archives being sent to rman_dcache every 30 minutes • PNFS metadata maintenance done automatically. CDF Taking Stock 2005-2006

  11. RMAN to SAN Experience in RunII • Test of SAN on bzora1 was a success. • We are going use SAN for keeper area • Practice recoveries from DCache. • Using SAN for D0 offline as well as D0 Luminosity • RMAN validation to determine backup file integrity • Recoveries from SAN were all successful CDF Taking Stock 2005-2006

  12. SAN issues • Current SAN is not 24 x 7 support • IDE disks used by SAN are not as reliable as other more expensive disks are. However, these have proven to be reliable. We do rman backup validate for backup files on SAN. Also recently recovery was done after restore from SAN. • Current SAN is trouble free except when the path failed a couple of months ago, and because the san is not dual path, it prevented backups over the weekend, as this is not 24/7 supported and we had to wait till Monday to get support. • Purchasing 24 x 7 SAN requires licensing and changes to O/S and dual path to FCC is not available. So keeping one on LOCAL and one on SAN. This reduces the risk. CDF Taking Stock 2005-2006

  13. SAM Schema • Production Deployments : Storage Location v6_1. • Work-in-progress - v6_0 ( SAM Request System) and v6_3 Retiring Files. • Upgrade to Mini SAM as SAM Schema Evolved. -> This facilitate individual developers to have copy of SAM metadata and seed data available for server software rewrite if needed. • Mini-SAM in Postgres. Initiative to move towards free ware Databases for SAM Proof of product not complete, requires testing with a dbserver  from the sam development team • SAM team will not push forward testing SAM API on postgres databases. CDF Taking Stock 2005-2006

  14. What’s Next ? • Upgrade cdfrep23 test bed replica to 10gR2. • Test SAM replication. • Deploy SAM replication in production as per justification. • ASO ( Advanced Security Option) Test. • Upgrade Designer and its repository to 10g • Bundling of Redhat renewal licenses into one P.O. • Cut New v6_0 and V6_3 of SAM schema into production. CDF Taking Stock 2005-2006

  15. Concerns • CDF should make sure to deploy 10g client to CDF software distribution. • Externally authenticated user db accounts should be converted to password authenticated. • Replication of SAM depends upon the stress test results on fcdfora4 and use case justification. • Simulation of Applications as we have for CALIB. Robust test Suite needed. • Single point of failure for SAM and DFC • Migration of DFC(BOOKS) to SAM . Plan and Schedule ? • Some of CDF Applications Data Model is not in Designer. • Any more Streams replica ? • Deputy CDF database Liaison ? • Special Clean-up jobs should be co-ordinated with css-dsg i.e. recalc of luminosity for DFC • Support for Day Light Savings Time change next year. CDF Taking Stock 2005-2006

More Related