Managing Production Systems: A Manager’s Guide to Prevent Firefighting.

1. Dennis Adams a s s o c i a t e s Managing Production Systems:A Manager’s Guide to Prevent Firefighting. 15 November 2006

2. Background • Different Systems / Vendors • VMS, TRU64, HP-UX, Ingres, Sybase, Oracle, Monitoring Tools. • Emphasis on IT Production • Support Administration, Sys Admin, DBA, Capacity Planning • Assistance to IT Production Managers • Deployment of Tools and Processes • IT Technical Architecture • Infrastructure Design and Architecture • IT Production Strategy • Global Standards • THERE ARE COMMON MANAGEMENT TECHNIQUES FOR RUNNING SUCCESSFUL IT PRODUCTION DEPARTMENTS • A Consultancy Service Dedicated to the Challenges of Managing IT Production teams.

3. Objectives of this Seminar • Identify some of the common challenges facing IT Production Teams today. • Highlight some of the pitfalls. • Introduce the MOPS Strategic Approach. • Explain how this can be applied in practice. • Highlight some “Action Points” to improve the way IT Production can be managed in future. • Some example brief Case Studies. • But first, some Technology…

4. University of Kentucky HP Superdome cluster • Four HP Superdomes • 256 total processors (64 procs per host/node) • Itanium-2 (Madison) processors rated at 1.25 TF sustained / 1.6 TF peak. • HPUX 11i v2 (11.23) • 2 Gigabytes of memory per processor • 7 Terabytes of total disk space • High speed, low latency Infiniband internal interconnect • Gigabit connections to public network

5. Compaq “Laptop” • The second Compaq luggable. • 20MB hard disk, 360K floppy drive, CGA screen. • A milestone in PC history. • It is the first of the clones.

6. HP 3000 • Advertised in • Computer Applications Pty • Australia

7. VAX 11/780 • CED Istituto di Radioastronomia 1981 • 1 MIP • 512 KB Ram • 64 MB Mass Storage

9. VAXMate • Intel 80286 • 1 MB RAM • 5.25” floppy disk • 20 MB 8” hard drive • Ethernet

10. Common Theme: The two different sides of IT • IT Development • Business Functionality • Speed of Delivery • Cost of Development • Development Projects may take months • Creating Competitive Advantage and ROI for the Business • IT Production • Reliability, Resilience • Stability, Scalability • Cost of Support & Maintenance • Production Support may be required over many years. • Delivering Day-to-Day Competitive Advantage and ROI for the Business IT Development and IT Production think and act differently.

11. Common Theme: Project Development and Deployment Development Production

12. Inside IT PRODUCTION Out of Disk / Table Space ? Investment? Oracle Upgrades Support Calls Application Failures Next Project... Planned Maintenance Data Copies Hardware Upgrades Backup Processes Extra Users = Extra Power & Storage

13. The Challenges facing IT PRODUCTION • Heterogeneous Data Centre • Lots of legacy applications to support. • More applications being thrown over the wall all the time. • IT Production can focus on “quick fixes”, rather than long term resolution of the underlying problems. • Lack of standardisation of production systems, leading to a broader set of skills needed to support it. • IT Production can be a “reactive”, rather than “pro-active”. • Difficult to justify future investment. • Threat of Out-Sourcing.

14. Dealing with the Challenge • There is no SIMPLE solution to the challenges of Managing IT Production • For IT production to succeed there should be Managers who are willing to “STEP BACK” and put in place a systematic STRATEGY. • Four Key Criteria for making IT Production • more Pro-Active, • more Client-Focused • a Strategic Managed Environment • in a better position to justify IT Infrastructure Investment • METRICS • OPERATIONAL TOOLS • PROCESSES AND PROCEDURES • STANDARDS M O P S

15. M O P S The “MOPS” Approach • The Basis for Pro-Active IT Production Management • Metrics • Identify Key Information to be captured and maintained in order to make informed Management decisions and justify IT Investment. • Operational Tools • Advise on the selection and integration of appropriate software tools to manage the infrastructure and the people who look after it. • Processes • Implement pragmatic processes based on ITIL and build linkages with Business and Development team processes. • Standards • Establish Technical Strategy and Architecture Standards to ensure that future applications can be supported at optimal cost.

16. M O P S Creating an IT Production STRATEGY • ANALYSE existing IT Production under the following headings: • Metrics • Operational Tools • Processes and Procedures • Standards • IDENTIFY the gaps • under each of these headings • PRIORITISE • from IT Production perspective, but also…. • ENGAGE with Sponsors and Business • Talk about what we are doing, and why • CREATE the Strategic Roadmap • owned by all stakeholders • INCREMENTALLY role out changes to the way the department works

17. METRICS M • “Some numbers which tell me what is happening” • You cannot control what you cannot measure. • Metrics can help us in several ways: • Help us understand where our resources are being used. • Identify which Applications are consuming our support costs. • Explain / Justify what we do to the business. • We must capture numbers that describe what IT Production does, in terms that make sense to the business. • This is NOT the same as Technical Performance Monitoring • “CPU % used” means nothing to the business • A SMALL no of basic Metrics ONLY. O P S

18. METRICS: Actions M • Create a database of the infrastructure, what it is, and the business purpose it exists for. • E.g. SERVERS by APPLICATION. • Capture the Time and effort your teams make. • Capture time by SUPPORTED APPLICATION, not by FUNCTION. • Your biggest budget cost is probably manpower – where is it being used? • For each APPLICATION, report on amount of time spent on support, no of support calls, • Report on Key Indicators to business. • “CPU Units” (not Percentage used !) • “Disk Units” (real allocation) • Highlight the “costs” of support of business applications. O P S

19. OPERATIONAL TOOLS M • Software Tools can assist in many ways: • Capture the METRICS on Time Used, Support Calls, Business Transactions, Infrastructure Supported. • AUTOMATE the day-to-day support role, Batch processing, Backup, Central Console, Alerting • The biggest danger with software tools is that they are implemented “in isolation” • Make sure that you have common naming conventions for all your applications etc. in different tools. • Reports can be cross-referenced. • Use a “referential” approach. • Remember that the more tools you have, the more complex the task of integrating them together. O P S

20. OPERATIONAL TOOLS: Actions M • Review what Operational tools you have. • Remove duplication of software • Exploit the capabilities of the software you already have. • Ensure that all software is integrated • Common naming conventions • A “referential” approach. O P S

21. PROCESSES & PROCEDURES M • Properly Implemented Processes can • Speed up the way support is managed • Reduce costs • Ensure that the business has a predictable service • Badly Implemented Processes can • Slow down the ability to respond to business needs • Increase Costs • Lead to a loss of Service • Introduce Processes that add Value, not Bureaucracy. • Concentrate on making it easier for external teams (developers, business units) to ask you to do things. • Be Pragmatic O P S

22. PROCESSES & PROCEDURES: Actions M • Look at the current procedures you have in place, compared with ITIL Standards. • Make sure that development teams, business units know how you work, and what they need to in order to get things done. • Put in processes to get involved with development projects at the earliest possible time in the project life-cycle. O P S

23. STANDARDS M • In some cases, the choice of Technology for a new Application can be driven by Developers’ Choice: • Useful Development Tools ? • Design and Development Features ? • Familiarity ? • The desire to try out the latest technology ? • Result: Applications whose Development costs may be Low, but the Support Costs may be high (even prohibitive). • Defining IT Production Standards can redress this balance. • Standards can contribute to controlling Costs of Maintenance & Support • Simplicity = Economies of Scale in Support O P S

24. STANDARDS: Actions M • Establish a Production Architecture Role • Collect and publish your Standards Capability • Make sure that the development teams get involved • Change the Development Project Management Methodology • Include Production requirements when projects are Initiated. • Have a process in place for looking at new technologies. • Define what you mean by “Production Ready” • Validate new products or technologies for “production readiness” and add them to the list (or not) O P S

25. SOME CASE STUDIES M Metrics Operational Tools Processes & Procedures Standards O P S How Does IT WORK in Practice?

26. CASE STUDIES (1) M • Infrastructure Support Teams did not know where the costs were going, and there were pressures on budgets. • METRICS: Introduce Timesheets into IT Production Team • Capture time which teams spent BY APPLICATION • Enabled us to identify the “problem applications” • Provided evidence for the Development teams. • Resulting in application changes • Reduced costs of IT Support. • Better visibility of costs. • Development Project teams became more “Support Aware”. O P S

27. CASE STUDIES (2) M • 30 to 40 DBA overnight call-outs to fix database problems. • OPERATIONAL TOOLS: Implement Alerts from Database Monitoring tool to central Operations Console. • This was a product that was already licensed to the client, but was not being used ! • DBAs advised of problems before they became incidents • Arrange pro-active outages to fix problems ahead of time. • Overnight work was not interrupted by incidents • Better quality of service to the business. • Number of call-outs was reduced - one or two per week. O P S

28. CASE STUDIES (3) M • Number of Software changes was high (every day). • Often changes were replaced by “patches” within a few days. • Significant support and implementation effort. • Business was not getting the service it wanted. • PROCESSES: Introduce Change Control Board and formal Change Control Processes, based on ITIL framework. • Number of changes reduced • Development Testing Effort was signficantly increased. • Post-implementation support effort reduced. • Business saw software changes implemented “right first time” O P S

29. CASE STUDIES (4) M • Business Development Proposal to introduce a new software package which was functionally rich, but would be impossible to support in production. • STANDARDS: Created a Production Architecture Team. • Engaged with Developers who were looking at this package, to emphasise the “Production Readiness” costs. • An alternative solution was chosen. • The costs of the proposed package in PRODUCTION could have been many times more • Although attractive to Developers, it could have resulted in massive financial loss to the company. O P S

30. CASE STUDIES (5) M • Various teams had incomplete spreadsheets listing machines they were supporting and what applications they ran. • METRICS: Initiated the creation of a centralised Configuration Management Database. • Servers: Location in Datacentre, Owner, Spec. • Applications: Server(s) • A Major outage resulted in the loss of one of the 6 datacentres • A recovery strategy was needed. • What Applications were important? • What Servers to restore first? • The information was available in minutes, instead of hours • Most of the services were restored within SLA. • Significant Financial Loss was reduced. O P S

31. Conclusions: How to MANAGE IT Production • Without a Strategic Approach, IT Production can become “Fire-Fighting” • Step back - Create an IT Production Strategy • The Key Elements for a Strategic IT Production Approach: • METRICS • OPERATIONAL TOOLS • PROCESSES AND PROCEDURES • STANDARDS • Approach: • Analyse your current activity against the “MOPS” • Identify and Prioritise work. • Implement Incrementally • Using appropriate management techniques, it is possible to run IT Production in a “Pro-Active” way, without “Fire-Fighting” M O P S

32. Managing Production Systems Time for any Questions? http://www.dennisadams.net email: dennis@dennisadams.net M O P S

33. Dennis Adams a s s o c i a t e s Managing Production Systems:A Manager’s Guide to Prevent Firefighting. 15 November 2006