Application and Performance Management Techniques for J2EE

1. Application and Performance Management Techniques for J2EE “Management Built In” for the Adaptive Enterprise Scott L. Williams HP scott.l.williams@hp.com

2. Agenda • Challenges: Testing, Management, & Repair of Applications • Embracing Management Throughout The Application Lifecycle • Management Tools for Operations • Instrumenting Your Applications For The Operational Environment • Demonstration of Application Instrumentation Tools • Call to Action

3. Challenges: Testing, Management, & Repair of Applications

4. 1. 2. 3. 4. The Ideal: Simple, Quick Resolution & Repair Requirements Gathering Design & Develop Staging Testing Production Application is “hanging” in production. Use OV to do “root cause” analysis and send results to developers. Use “root cause” analysis results to narrow down which software contains bug. Fix bug. Deploy bug fix to staging. Execute regression tests to verify integrity of bug fix. Release bug fix to production.

5. the application error occurs No Shared “Context”  Difficult For IT/Ops To Correct Problems Application Development IT/Operations No way to help developers find error within the application!

6. Examples of common pain points today • Application integration (many technologies, techniques, etc.) • Poor communication (business+dev+operations) • Deployment failures (insufficient information) • Difficulty configuring application security (technology, process) • Maintenance (again, insufficient shared information)

7. Most time and money is dedicated to maintenance in production IT Today Innovation 10% 30% Consolidation/ Integration 60% Operation Source: HP IT department

8. Removing The Barriers To Success • Free up developer resources • Today: developers often spend too much time supporting their applications when those applications degrade or fail in production • Goal: spend less time supporting applications and more time creating new ones (greater alignment with new business needs) • Free up IT resources • Today: Large numbers of operations staff; manual processes; “archeology” of poorly documented applications • Goal: Well documented apps; automated troubleshooting; better management at all levels (infrastructure, apps, business processes) • Drive complexity out of the software lifecycle • Today: Hidden assumptions, poor documentation, “silos” • Goal: Meta-data (documentation); “management built in” to applications; context shared throughout entire lifecycle (business-development-operations communicate);

9. Adaptive Enterprise design principles Simplification Simplify applications and systems to reduce complexity and risk. + • Applied consistently across: • Business processes • Applications • Infrastructure Standardization Standardize the way IT assets are used with common components and processes. + Modularity Improve performance by managing infrastructure components discretely or collectively. + Integration Easily manage and modify the environment through a uniform system of relationships.

10. What is “Management for the Adaptive Enterprise?” • Do more with less • Remove barriers to success • Shift IT Investment from maintenance to innovation • Be ahead of the competition • Be competitive or be outsourced

11. Moving from maintenance to innovation 3 years from now Today 10% 40% 30% Innovation 60% 20-25% • In large enterprises up to 75 % are personnel or services costs • Hardware, software and infrastructure only 25 % Consolidation 35-40% Operation Source: HP IT department

12. Embracing Management Throughout The Application Lifecycle

13. the application the application error occurs error occurs Application Development IT/Operations No way to help developers find error within the application! Shared “Context”  IT/Ops Can Correct Problems Application Development IT/Operations CONTEXT: instrumentation points, performance metrics, app meta data, etc. Use “context” to communicate problem back to application developers

14. track track track monitor monitor monitor control control control Start Here

15. Roles:Who Manages Apps in Production? • Application “Owners” (business sponsors) • Customer Management • Renegotiate Contracts • Measure/Monitor Compliance with SLAs • IT/Operations (traditional, platform management) • Ensure health of hardware, OS, some middleware • 1st line of defense when problems occur • Application Managers (newer concept) • Ensure health of application containers and custom applications • 2nd line of defense. Can often alter/control/repair without needing dev support. • Software Developers • Last line of defense. • When all else fails, they use all runtime context to debug & repair problems

16. Market Observations & Trends – Gartner App Responsibility for app • Best-of-breed is most popular strategy. • Application development & support are most often responsible for application management. • Custom applications are most need of management • “Gartner believes …that responsibility for application management increasingly resides outside of the traditional IT operations organizations”. development management IT operations 40 App support 30 LOB % 20 10 outsourcer 0 other Group • Gartner “Application Management Poll Reveals New Trends” April 15, 2004

17. BSM AM Whom HP OpenView Application Management Targets Today CIO LOB Owner Biz/IT Liaison IT Controller LOBFinance Bus ProcOwner App Manager IT Ops Manager Infra Svcs Manager App Support Network Mgr ServiceDeskMgr ServerMgr App Dev App QA

18. Software Developer Tools: “Enterprise debugger” IDE Architecture Models Application Owners (business sponsors) Tools: “BPM/BAM Dashboards” CRM/Service Desk enterprise “coredump”: call stack traces performance info audit trail business runtime: SLO’s, KPI’s, etc. business processes Business, Application & Environment Context application runtime: JMX, WMI transaction traces, etc. operational runtime: mgmt events, infra. “health”, etc. Operations/IT Tools: Performance Monitoring Operational Status Monitoring Infrastru. Troubleshoot Tools Application Management Tools: Application Troubleshooting (JMX info, etc.) Service Management Tools

19. Management Tools for Operations

20. 3 Pillars of Management • Tracking • observe aspects of a single thread across multiple components. • technologies: logging & auditing • Monitoring • capture runtime and historical events of a particular component, for the purpose of reporting and notifications. • technologies: Real-time statistics (JMX), logging, & auditing • Control • alter the behavior of the managed component directly without recourse to interrupt the operation of the component. • technologies: JMX, custom techniques

21. What is “Tracking?” • Measurement of “operational performance” • Combination of JVM profiling & application logging • Often is “durable” recording of what occurred. • Like “frames in a movie” • Some tools could “play back” the recording. • Helpful during testing & staging

22. What is “Monitoring?” • Measurement of “business performance” (SLAs, KPIs) • Measurement of current “operational health” and “business health” • Much “context” (operational & bus) only known by applications: so use application logging & JMX • Some is “durable” recording of what occurred. • Some is transient or “current snapshot” • Management Events & Corrective Triggers

23. What is “Control?” • Automated changes or repair of application. • Manual changes or repair to application. • Often in response to management events “fired” during monitoring. • Sometimes modifies operational aspects of applications. • Sometimes modifies business aspects of applications. • Integral to building an adaptive enterprise and adaptive applications. • J2EE Applications often expose JMX MBeans for this purpose.

24. Management Products for “Tracking” • HP OpenView Transaction Analyzer • HP OpenView Internet Services – measures availability (and response-time) by creating synthetic transactions (aka “robots”) • Borland ServerTrace • Actional (web services) • JVM-level Profiling Tools • Etc.

25. Management Products for “Monitoring” • HP OpenView for Operations, Reporting, Graphing • HP Service Desk • HP OpenView Internet Services+ OVTA(real end user response times and volume metrics, essential for SLA monitoring -- e.g. response-time < 5 secs AND volume < 10,000 TPS) • BMC • CA Unicenter • Microsoft Operations Manager & Performance Monitor • Actional (web services) • Etc.

26. Management Products for “Control” • HP OpenView for Operations • BMC • CA Unicenter • Microsoft Operations Manager • Etc.

27. Instrumenting Your Applications For The Operational Environment

28. What should you manage? • Think in terms of an application’s “management interface” • Use SLAs, business requirements, & IT/operations needs to help you decide what management to support • Management is a combination of what can be done at the platform (JVM, J2EE Containers, etc.) & what you need to code • You only need to code context that isn’t available from the platform: (business-specific context, operational statistics not gathered by platform, etc.)

29. Customers Marketing, Analysts, Architects Solution Architects, Industry Trends, etc. IT Operations business management contracts with end users, service level agreements, etc. operational management starting-point management “best practices” management requirements operational management new requirements learned over time through experience

30. IT Operations determine “dials” and “knobs” for mgmt management tools manage… are specifications for Management Interface of Solution management requirements communicate… are specifications for Solution (business logic, data, etc.) functional & systems requirements

31. IT Management Interfaces “HR” App “Call Center” App Management Interfaces Management Solution(s)

32. Tracking • What To Do: • Logs and trace for method “entry” and “exit.” • Performance metrics (at JVM-level as well as app container level) • These are like frame-by-frame recordings of program executions. They are durable • Available Technologies & APIs: • JDK 1.4 or later: use standard java logging • JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport ) • JVMTI (JVM Tooling Interface) for tracking/monitoring tools

33. Monitoring • What To Do: • Logs and trace for operational as well as “business” statistics. • Business audit logs for historical business data (store exchanged business documents, business events, etc.) • JMX and WMI for application instrumentation • Performance metrics (at JVM-level as well as app container level) • Available Technologies & APIs: • JDK 1.4 or later: use standard java logging • JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport ) • JMX (MBeans) – “custom” MBeans as well as container-provided

34. Control • What To Do: • react to predictive “triggers” and actually change running servers & code • modify the behavior of servers & code “in flight” without shutdown • solve application problems/issues without always requiring developers • Available Technologies & APIs: • Custom-coded “control point” management objects using JMX (MBeans) • Custom-coded “control points” using: config files, web services, RMI, etc. • Container-level “control points” accessible via JMX, config files, etc. • React to change of state of “control point” objects

35. Best Practices for coding mgmt interfaces • Use JMX for operational and business “health” & as “control points” • Use JMX to provide “rollups” such as: • averages over time • business-specific stats: # of “high value” customers this hour • current operational load on system • current business activity rollup: $ processing right now, etc. • MBeans are for monitoring & “control”: Change their state to control the application.

36. Averages & Roll-Ups Belong Outside of application’s source • Could provide averages over time: “transactions per hour”, “high value customers served today”. • But!, better to put these calculated metrics outside of the application and outside of MBeans: • Management system configurations (OVO templates) • Management “rules” engines, event engines, etc. • Provide simple, atomic data through JMX. Aggregate it externally.

37. Don’t count on “native” JMX Management Events • With JMX you can subscribe to and generate events. • JMX events are “seen” within the JVM. But aren’t sent outside to the management system. • Different management products have their own event mechanisms. And WSDM (Web Services Distributed Management) standard has another. • If you will be in a heterogeneous environment with many technologies, don’t use JMX or WBEM notifications. • Consider not using “native” JVM management events

38. A Very Simple! Standard MBean • public interface MoviePreviewStatsMBean { • public int getPreviewDownloadsCount (); • } • public class MoviePreviewStats implements MoviePreviewStatsMBean, java.io.Serializable { • public MoviePreviewStats () {} • public int getPreviewDownloadsCount () { • return m_nPreviewDownloadsCount; • } • private int m_nPreviewDownloadsCount = 0 • protected void resetPreviewDownloadsCount () { • m_nPreviewDownloadsCount = 0; • } • protected void incrementPreviewDownloadsCount () { • m_nPreviewDownloadsCount++; • } • }

39. Different “Flavors” of MBeans • Standard MBeans • Useful when the structure of managed data is well defined and unlikely to change • Named with “MBean” added on the interface • Attributes with getter/setter methods • When to use: When first using MBeans; when the management interface isn’t changing much. • Dynamic MBeans • must implement “DynamicMBean” interface • is like DII in CORBA – your MBean doesn’t need a method signatures that match how the outside world “sees” it • You supply the “MBeanInfo” meta data for the bean • When to use: When you want a level of indirection between the signature of your MBean and the actual implementation; When impl side is dynamic (at runtime as well as design often changing). • Model MBeans • (A subtype of dynamic MBeans) • “RequiredModelMBean” always available and implements the “ModelMBean” interface • The infrastructure implements most of the state caching, methods, etc. you would normally write yourself. • When to use: when you want to configure MBeans from outside of the source code (like from XML file)

40. Publishing Your MBeans • If you want the outside world (management systems) to see your MBeans, you “publish” them to an MBeanServer. • Each MBean that is published is like a “pointer” to an actual Java object. • Consumers of MBeans never “touch” the actual MBeans. Instead they ask the MBeanServer to talk to the MBeans. • Some Best Practices: • As often as possible, use the standard JMX API for this (for instance, BEA lets you use an MBean EJB) • Use unique “domain” names for each application.

41. Code Sample for Publishing an MBean /** * Create the MBean. Our application will be responsible for * updating its value throughout the running of the application. */ Employee employeeMBeanRef = new Employee (); /** * Access the MBeanServer. */ MBeanServer mbeanServer = MBeanServerFactory.createMBeanServer(); /** * Create “object name” for Bean. Publish it into the MBeanServer, using * "HR_domain" as the domain part of the MBean's name attibute. */ ObjectName beanName = new ObjectName("HR_domain:Name=Employee&type=manager"); mbeanServer.registerMBean( employeeMBeanRef , beanName);

42. Demonstration of Application Instrumentation Tools

43. SPI Configuration: user defined metrics JMX Metric Builder edit configuration file your application HP OpenView for Operations MBeans (the management interface) JMX SPI Container MBeans J2EE Container

44. Call To Action • Think about app instrumentation “holistically” rather than as some APIs. (Think about the “management requirements.”) • Work within a development lifecycle that embraces “manageability built in” as fundamental to success • Determine your management requirements and management interfaces up front. • Build your applications to support: tracking, monitoring, & control. • Use technologies such as JMX, web services & SOA to help you create flexible & robust applications for the adaptive enterprise.

45. Questions