Discussion on tasks and deliverables

1. Citi Upgrade to ControlCenter 6.1 Preparation MeetingsCiti Engineering - Oct 19, 2009Citi Operations – Oct 20, 2009 Discussion on tasks and deliverables

2. Agenda - Oct 19th • EMC Deliverables • Script to resolve Host naming issues • Script to review the discovery relationships • Test-run validation on the Repositories • Design change recommendations • Citi Engineering Deliverables • Citi platform agent upgrade utility • Citi upgrade process documentation • Upgrade to CC 6.1 project plan • Datacenter rollout schedule and special requirements • Citi roll-out dependencies • Upgrade teams • Pre-CC 6.1 Agent upgrade schedule and requirements • General discussion to make this a successful upgrade

3. Agenda – Oct 20th • Review the CC 6.1 Upgrade plan and responsibilities • Agree on design changes • Agree on rollout schedule for Infrastructures and datacenters • Agree on agent upgrade process and rollout • Agree on the pre-upgrade work needed by EMC to be done • General discussion to make this a successful upgrade

4. EMC Deliverables – Discussion Summary • Script for resolving bad host domain names - Completed • Script to printout the MO Access Tables – In Development • Staging of each datacenter for test-runs – Done • Datacenter DMP imports – Only 3 DMPs received • Test CC 6.1 upgrade fro each datacenter • Design review and recommended changes

5. EMC Scripts FQDN Script • Will mine the Repository • Change Domain Names associated with host names • Notes: FQDN Scripting; Citi using incorrect binding, cgt.ini should stay set and will fix the binding of the agent to IP address; need to follow industry standards of host file; hosts need to be correctly placed in host file; most of what John M. sees is host DNS – can’t trust what is in the host file; without changes to citi procedure this will be an ongoing problem for Citi – duly noted, per Citi • Duplicate Host names • Procedure on what to look for • Delete Host script to whack the unwanted host in the repository • Notes: multiple entries may not be discovered; would john b want notification script to indicate when there are differences in naming? Will consider. Al wants to know how they can mitigate naming procedure problems with their customers in their reporting? Jim: RQP or ER to ask for long-term resolution to problem. Vivek will take up ownership of pursuit of this long term solution from ECC. Production can query for duplicate hosts. Joe: splitting piece is fraction of overall agent challenges. Looking to tweak #2 script for 6.1 (now 5.2 and 6.0) – will look into preserving what’s there. • MO Discovery Read-out Script • Dump the MO Access Tables so discovery relationships can be reviewed • Can now see the readout information via script ENG developed – provides all information on how to rediscover databases; John B. requested to include home, base, listener port, (John B. notated hard copy.) Jim will review John’s requested refinements; this information would then go back into the bulk loader utility; could automate still more; shows who is bound to what agent;

6. Notes, cont. • MO Discovery Bulk Loader • Facilitate the with an auto-loader the discovery of databases • Script to remove the Store from the Server/Repository • Part of recommended design changes

7. Pre-Upgrade Validations • EMC has staged a VM for each Datacenter • DMP files will be imported notes: jim will validate these procedures in his lab • Only 390, Rutherford and Southwest have been provided • EMC will test all scripts against the staging VMs • FQDN Script • Duplicate Host Name resolution procedure • MO Access read-out script • Clean-up of the Store on the Server/Repository • EMC will run a Repository 6.1 Upgrade • Validates that no issues will exist in the repository portion of the upgrade • EMC will address any issues found • Notes: Upgrade to 6.1; upgrade to UB5; will churn the repository several times; pre-work done on 6.0; backup and restore are in project plan, but Jim will not be able to run against it in his lab; Citi has opened up code and changed it, prohibiting bura activity; April 15th was when it was released to Citi with documentation;

8. Design Recommendations • SAGS converted from Solaris to Windows • SAGS for Vmax is first • 7.1 need RPQ approval for any Sybase on Solaris – it will work but need to validate for coverage; should submit RPQ for Sybase on Solaris 7.x; AIX is covered in 7.1; SE needs to go to 15.0.3 or need SE RPQ to greenlight 15.0.2; Vivek will drive the RPQ; JIM to confirm whether a newer version of SE can speak with an older version of SE; Scott Shaffer<<<<<<HERE’S where I left off as the deck had been updated as Jim showed>>>>>> • May need add’l memory to enable concurrent user session to with1.5-2gb; Citi has 4 gb memory on server; this is not for remote configuration mode; Citi noted Tom R.’s comment about running a client app on a client server; the Citi logon script will put on a workstation script, etc. – can’t be done per audit and control rules; application server mode that has to be licensed; admin mode is not the correct one to use. Console performance is now a problem; does grayscreen, takes too long to load; discussion of using Citrix vs. remote desktop; view would be lost w/Citrix; need to discuss further …. • 2 centeras to manage; sizing exercise needs to be completed; need to accurately project growth over next 12-24 months and tech refresh; Al M.: last 3 years haven’t seen any significant growth; also need # arrays, MO’s, db’s, etc. and considered on a per-data center basis; Al: last year added 1.5 PB, with most growth in SW Datacenter; now could consolidate on the new VMAXs that they have gotten; NAS, adding Celerra; will not manage Netapps, however. But, as Al M. points out why should they manage two different products down the line? Jim mentioned that scottshaffer had said looking to address in SRM7; Jeff indicated management of over 1 billion files and not managing queue trees were two problems preventing it being done over a year ago; their existing celerra report is custom, in that it is a tweaked canned report (custom query.) • Jim rec’d dedicated WLA archiver VM’s – will recommend optimal number and should be easy to incorporate earlier rather than later; very quick install, can start collecting data and then work into use; Al wants to do after 6.1 • Stores on the Server/Repository to be shutdown and deleted • EMC will provide a script to remove the store from the console • All Store VMs be set to 2gb of vMemory • Add a VM for the Store removed from the Server/Repository • All Store VMs be set to 2gb of vMemory • StorageScope VM be set to 2gb of vMemory • Dedicated WLA Archiver VMs • SAN Discovery midwest and Georgetown are all cisco; others mixed • Number of FCC Agents Jim: agent servers can all resolve on a 1:1 • Mixed SAN Vendors – number of and location SMI Agent Provider • Complete ownership of ControlCenter Infrastructure and agent Servers

9. Citi Deliverables – Discussion Summary • Citi Wrapper for UNIX • Citi Wrapper for Windows • Citi agent clean-up scripts • Citi Upgrade documentation • Infrastructure • Agents • Solutions Enabler Upgrade Provide the details

10. CC 6.1 Upgrade Plan- Discussion Summary • Each Datacenter sequentially • Review upgrade cycle • Review of Citi requirements • Upgrade teams

11. Upgrade Plan – High Level • Upgrade cycle for each datacenter, one at a time, starting with 390 • Week 1 - For upgrade of infrastructure • Including any final pre-upgrade adjustments • Custom Reporting additions • All Infrastructure components upgraded • Start critical agent upgrades • Validate Citi Wrapper on production servers • Week 2 - For continued critical agent upgrades and discovery validations • Test reporting information • Week 3 to ? - Other Agents upgrade and monitor for stability • Validate reporting information • Validate SAN discovery • Validate Performance Manager information

12. Citi Datacenter requirements • Critical Reports generated on the 20th of every month • No changes 5 days prior • What are the blackout dates for Citi • Monthly • Yearly • Discussion on all other requirements to be aware of and worked around

13. Upgrade Teams • First Upgrade – 390 • Done by EMC as the lead (Ray & Jim) • Second Upgrade • Done by Citi as the lead (Joe & Murphy) • EMC Coaches • Remainder of the Upgrades – discussion topic • Citi to decide how to proceed • Citi does the remainder • EMC does the remainder

14. Upgrade Project Plan Structure • First Draft of the project plan, not all details are established • Identify all tasks • Task durations and dependencies • Add “black-out” dates

15. Notes from meeting review of project plan • Start date of 11/1 was based on Vmax getting into production during same timeframe; stated as possible when we met in august; now looks more like getting started 12/1 • How to do: 2 scenarios / flexible • Citi traditional process • EMC do first with Citi watching; Citi drive next w/EMC watching • Reviewed MPP onscreen; tasks, assignments, etc. • Bulk of EMC work at start • Discussed packaging and what the SA’s will have to do • Was email notification being eliminated? (Joe): Citi will talk offline and let Citi Eng know. • Joe: how can he save two servers by un/re-installing powerpath; problem is bad SAN stack • Vivek might take week or two to fix this problem as part of the overall project • 11/20 billing date at Citi; freeze after 11/15  December 1st next best opportunity for upgrade • #1 John B needs to verify if the freeze guidelines came out: 12/15-1/15 across the board at Citi is global freeze; some sites start earlier (before Thanksgiving); Southwest: a few arrays, do start early – but rest fall in the 12/15-1/15 freeze. • Cannot touch infrastructure, applications, ecc, etc. NO CHANGES. Variances very difficult to get. • If a DC was done during the freeze time, could not touch any agent on any production server during freeze • Jim discussed option to start in non-secure mode that is part of 6.1; hasn’t tested per Vivek; would permit getting a jump start; UB7 can turn infra secure or unsecure automatically • Vivek to test 6.1 agents with 6.1 high watermark DCP; all OS; various stack levels; 6.1 infra w/6.0 agents; various san stacks • John B. commented to Vivek that engineering is ‘way more prepared than you were in 6.0” • Vivek confirmed he’ll execute the dry run and validated installation guide • Need to validate data out once the environment is flipped • Both production and engineering at citi need to do their validation review • Jim: you can take snapshots as you roll through the processes; but validation is when you bring it up. All steps = approx three days • Discussed important benefits of UB7 vs. UB5 • Is there a line for sizing the repository (1.15 + 7gig) and add’l temp storage required • Vivek: for b/u appx storage Citi rec’d is 4 times, per Vivek

16. Summary of Pre-Upgrade Tasks

17. Custom Reporting Tasks

18. Staging for the upgrade • CSE Lab is staged with a VM for each Citi Datacenter • Only 390, Rutherford and Southwest have been provided • Provided DMP’s have been imported

19. Validation Phase (390 DC VM) • Run a simulated CC 6.1 upgrade • Validates no issues in the repository ONLY

20. Validation Phase (other DC VMs)

21. Architectural Review and Recommendations • Recommendation for • Additional Servers • Additional VM’s • Changes to exiting VMs • Additional functionality • SAN Discovery • Performance Manager

22. MO Discovery and DCP validations • Review the discovery relationships and determine corrective actions

23. Store Alert Issue • Alert driven by core calculations of 2GB of memory • Adjust VM with Store to 2GB of vMemory

24. Upgrade per Datacenter

25. Questions