Elicitation and Utilization of Application-level Utility Functions

1. Elicitation and Utilization of Application-level Utility Functions Paul de Grandis and Giuseppe Valetto Drexel University - Dept. of Computer Science

2. Outline • Motivation • Goal • Approach • Engineering considerations • Experiments • Conclusions

3. Motivation • Autonomic provisions are often costly and complex • Many times, those provisions target only specific classes of problems • expected or likely conditions • It’s not only a matter of requirements … • … it’s also hard to develop complete system models that enable general-purpose adaptations

4. The challenge • Unexpected conditions are often the most critical • Typically occur only on the field • By definition, they have not been accounted for by designers, programmer, testers etc. • Potentially devastating for the system • One could argue that autonomic provisions fulfill their promise when they can handle the unexpected

5. Designing for the unexpected

6. Our goal • Enable non-analytic self-assessment of SW applications • Self-assessment: map the state space of the application to some abstraction of value • Recognize where the system on in that map • Non-analytic: do not require a complete model of the system in order to enable self-assessment

7. “bad” state “good” states Two-steps strategy Self-adaptation Self-assessment ? State space

8. Approach: application-based utility

9. Utility of an application • What is the app. value for, e.g., its client? • A single scalar: Uapp [0,1] • Utility function: Uapp = f(V[s0,…,sn]) • s0,…,snare state variables • Network measures • System measures • Application specific measures • User-observable measures

10. Our vision • Elicitation: construct Uapp for a given application • Often, for a given task or feature • Embedding: insert runtime code to monitor state variables and compute Uapp • Utilization: continuously evaluate Uapp • We are interested in a method that can handle legacy applications as well as new ones

11. Our process • Identify utility from the client perspective • Instrument application and environment • Run experiments and observe V[s0, …,sn] • Synthesize Uapp(V[]) • Substitute Uappin the instrumented code • Manual, expert knowledge • Automated • Automated • Off-line, statistical correlation • Automated

12. Observation and synthesis Statistical correlation Application Environment • We operate at multiple distinct levels • Typically, application- vs. environment-level • Collect lots of lower-level monitoring data from the running environment … • … and relate it statistically to some externally visible manifestation of application goals

13. Example – FTP service • Simple protocol-based application • Utility is straightforward: • For an FTP client, utility = client throughput • Relevant environment (socket-level) parameters: • Latency • Bandwidth • Loss rate • …

14. Eliciting application utility U Applicationlevel t Environmentlevel

15. Embedding application utility U Applicationlevel t Environmentlevel

16. Engineering approach

17. Elicitation and embedding framework • Aupy: “Application Utility in Python” • Framework leveraging meta-programming capabilities of Python to instrument applications • Two flavors: • Python annotations (legacy code-friendly) • Python contexts (granular and flexible) • Open-sourced at: http://www.aupy.org

18. Aupy annotations • Added to method definitions • @monitor(foo) bar triggers evaluation of utility function “foo” when “bar” is executed • u.f.method declarations: • @pre-utility foo • @post-utility foo • @utility foo • @concurrent-utility foo • foo() can contain monitoring code (elicitation) or code computing Uapp (embedding)

19. Aupy contexts • Contexts are Python scoping constructs • “with” keyword establishes a custom scope • Aupy uses them to associate utility functions to arbitrary blocks of code • Finer-grained control • You can pass to a context utility function, arguments, configuration data, callback information …

20. Benefits of contexts • Contexts (and hence u.f.) can be composed • Multi-step u.f. • Utility of a u.f. • Support of selection logic among multiple alternative u.f.

21. Experiments

22. FTP experiment set-up • In-the-lab data collection • 125 test runs using dummynet to throttle network • create a uniform map of environmental conditions • Two download scenarios: • Bulk-ftp: transfer one large file • Small-ftp: transfer multiple small files • Off-line data analysis and u.f. synthesis

23. FTP results • X=inter-arrival time • Y= socket throughput • Very different u.f. for the 2 scenarios • We plugged in elicited u.f. • kept monitoring client throughput online and compared

24. FTP-bulk results Estimated Actual

25. Another experiment – log explosion • Real-world case study • Logging component of a Python-powered popular WWW site gone awry • Explosive growth of log would block the WWW site in a few seconds • Even prevent remote login from sysadmins • Could we apply non-analytic self-assessment to this problem?

26. Log explosion experiment • Phase 1: catching log explosions • Utility of WWW site drops almost immediately to 0 upon log explosion • need to quickly capture acceleration peaks in disk writes per process • before disk resource is exhausted • Phase 2: preventing log explosions: • hypothesis: utility is Uapp ≈ f(1-pnboom) • pboom : probability that a log explosion is going to happen “some time soon”

27. Log explosion results • We were able to catch log explosions • Captured log explosion dynamics for disk writes • Write speed raises 2 orders of magnitude each second • Uapp looks like a step function • Collapse occurs in a few seconds • But we could detect the front and turn off logging in time • The prevention problem is more complex • Would require very frequent sampling of disk write speed • That interferes too much with normal operation • We cannot really “see” preliminary signs and estimate “pboom”

28. Conclusions

29. Lessons learned • We have understood the elicitation process • Repeatable • Requires only a modicum of domain knowledge • We have acquired know-how on the mechanisms for elicitation and embedding • Can be made minimally intrusive • Can be applied to legacy systems • Layered approach is generally valid

30. Lessons learned • We have insight on the task-based nature of application utility • Need an array of u.f. that can be selected opportunistically • On a per-feature basis • Also based on different tasks • Idea applies especially well to: • Protocol-based applications • Infrastructure-level services

31. Research directions • Towards in-the-field data collection • Deploy Aupy-instrumented application • Multiple features are exercised • Assign clusters in the monitored space to exercised features • Elicit all feature-specific u.f. from the clustered areas

32. Research directions • Towards automated utility definition • Use cooperative negotiation and dependency discovery to figure which application characteristics to consider as utility • Leverage utility map to guide adaptations • Reinforcement learning to compute trajectories back from bad utility to good utility states • Actionable trajectories: can be traversed by tuning some state variables • Utility delta as RL “reward”

33. Questions?