Towards Autonomic Hosting of Multi-tier Internet Services

1. Towards Autonomic Hosting of Multi-tier Internet Services Swaminathan Sivasubramanian, Guillaume Pierre and Maarten van Steen Vrije Universiteit, Amsterdam, The Netherlands.

2. Hosting Large-Scale Internet Services • Large-scale e-commerce enterprises use complex software systems • Sites built of numerous applications called services. • A request to amazon.com leads to requests to hundreds of services [Vogels, ACM Queue, 2006]. • Each site has a SLA (latency, availability targets) • Global optimization-based hosting is intractable • Convert Global to per-service SLA • Host each service scalably. • Problem in focus: Efficient hosting of an Internet service.

3. Web Services: Background Service Response Service Y Service X Service Req. Service Req. (XML) Application Server DB Queries DB Service Response (XML) DB Response e.g., DB2, Oracle, MySQL E.g., JBoss, Tomcat/Axis, Websphere • Services – Multi-tiered Applications • Perform business logic on data from its data store and from other services. • E.g., Shopping cart service, Recommender service, Page generator. • Exposed and restricted through well-defined interfaces • Usually accessible over the network • Does not allow direct access to its internal database

4. Service X Scalability techniques applied to service hosting Useful for compute-intensive services (e.g., page generators) Service Y Application Server Application Server DB Application Server Application Server

5. Service X Scalability techniques applied to service hosting Reduces load on application (if hit ratio is good) Service Y Application Server Response Cache DB Response Cache Cache service Responses

6. Service X Scalability techniques applied to service hosting Reduces DB load (if hit ratio is good) Service Y Application Server DB Caches DB DB Cache Cache Query Results e.g., IBM’s DBCache, GlobeCBC

7. Service X Scalability techniques applied to service hosting Service Y Response Cache Response Cache Response Cache Application Server DB Reduces response time Useful if other service is across WAN or does not meet SLA

8. Response Cache Response Cache Application Server DB Response Cache Application Server DB Cache Response Cache DB Cache Scalability techniques applied to service hosting • Resource provisioning for a service • Wide variety of techniques at different tiers to consider • What is the right (set of) technique(s) for a given service? • Depends on: locality, update workload, code execution time, query time, external service dependencies • Too many parameters for an administrator to manage! • Can we automate it (at least to a large extent)?

9. Autonomic Hosting: Initial Objective “To find the minimum set of resources to host a given service such that its end-to-end latency is maintained between [Latmin, Latmax].” We pose it as: “To find the minimum number of resources (servers) to provision in each tier for a service to meet its SLA”

10. Proposed Approach • Get a model of end-to-end latency • Lat = f(hrserver , tApp, hrcli , tdb , hrdbcache ,ReqRate) • hr = hit ratio, t = execution time • f – Latency modeling function • Little’s law based network of queues • MVA (mean value analysis) on network of queues • Or other models?

11. Proposed Approach (contd..) • Fit a service to the model • Parameters such as execution time can be obtained • Log analysis, server instrumentation • Estimating hr at different tiers is harder • Request patterns and update patterns vary • Fluid-based cache models assume infinite cache memory • Need a technique that predicts hr for a given cache size

12. Virtual Caches • Virtual cache (VC) – means to predict hr • Cache that stores just the meta-data [Wong et.al., 2002] • Takes original request &update stream to compute hr • Smaller footprint • Can be added in different tiers such as App servers, Client stubs, JDBC drivers. • What will be hr if another server with memory d is added to a cache pool with M memory? • Run a VC with M+dmemory • A VC with M-dmemory gives hrwhen a server is removed. • Running VC for distributed caches • N caches servers, each with M memory • Run VC in each server with M + M/N memory • => Avg. hr when a new server is added

13. Resource Provisioning • To provision a service • Obtain (hr & t) values from different tiers of service • Estimate latency for different resource configurations • Find the best configuration that meets its latency SLA • For a running service • If SLA is violated, find the best tier to add a server • Switching time? • Addition of servers take time (e.g., cache warm up, reconfiguration) • Right now, assumed negligible • Need to investigate prediction algorithms

14. Current Status & Limitations • Goal: To build an autonomic hosting platform for Multi-tier internet applications • Multi-queue model w/ online-cache simulations has been a good start • Prototyped with Apache, Tomcat/Axis, MySQL • Integrating with our CDN, Globule • Experiments with TPC-App -> encouraging • Experimented with other services • Current Work • Refining Queueing Models for accurate latency estimation • Investigating availability issues

15. Discussion Points • Utilization based SLAs • Other prediction models • Does cache behavior vary with req. rate? • Failures • How to provision for availability targets? • Multiple service classes

16. Availability-aware provisioning • Availability-aware provisioning • To provision for a required up-time • Must consider MTTF and MTTR for servers in each tier • Caches have different MTTR than AppServers • How to provision? • Strategy 1 • Perform latency-based provisioning. • For each tier, add additional resources to reach target uptime • Strategy 2 • Formulate as a dual-constrained optimization problem.

17. Dynamic Provisioning • For handling dynamic load changes • Need to predict workload changes • Allows us to be prepared earlier • Adding/reconfiguring servers take time • Prediction window should be greater than server addition time • Load prediction is relatively well understood • Prediction of temporal effects?

18. Thank You! More info: http://www.globule.org Questions?