Adaptive Static Resource Optimization

2. Adaptive Static Resource Optimization David Wei and ChanghaoJiang The AJAX Experience 2009 Sep 15, 2009 Boston, MA

3. Best practice: from theory to operation? “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image

4. Best practice: from theory to operation? “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image Interface & Infrastructure Operation

5. Agenda

6. Optimization must be adaptive

7. Facebook’s day-to-day challenges • Deep Integration • Viral Adoption • Agile Development • Performance optimization has to be adaptive!

8. Perf optimization must be adaptive • Day 1: Some smart engineers start a project! <Print css tag for feature A> <Print css tag for feature B> <Print css tag for feature C> <print HTML of feature A> <print HTML of feature B> <print HTML of feature C> … “Let’s write a new page with features A, B and C!”

9. Day 2: Some smart engineers run YSlow and says… <Print css tag for feature A> <Print css tag for feature B> <Print css tag for feature C> <print HTML of feature A> <print HTML of feature B> <print HTML of feature C> … “A & B & C are always used; let’s package them together!”

10. Day 2: Awesome! <Print css tag for feature A&B&C> <print HTML of feature A> <print HTML of feature B> <print HTML of feature C> …

11. Day 3: feature C evolves… <Print css tag for feature A & B & C> <print HTML of feature A> <print HTML of feature B> If (users_signup_for_C()) { <print HTML of feature C>} …

12. Day 3: <Print css tag for feature A & B & C> <print HTML of feature A> <print HTML of feature B> If (users_signup_for_C()) { <print HTML of feature C>} … A&B are always used, while C is not. ..

13. Day 4: feature C is deprecated <Print css tag for feature A & B & C> <print HTML of feature A> <print HTML of feature B> // no one uses C { <print HTML of feature C>} …

14. Day 4: we start to send unused bits <Print css tag for feature A & B & C> <print HTML of feature A> <print HTML of feature B> // no one uses C { <print HTML of feature C>} … It is hard to remember we should remove C here.

15. One months later… <Print css tag for feature A & B & C & D & E & F & G…> if (F is used) <print HTML of feature F> <print HTML of feature G> if (F is not used) { <print HTML of feature E>} … Thousands of dead CSS rules in the package.

16. One months later… <Print css tag for feature A & B & C & D & E & F & G…> if (F is used) <print HTML of feature F> <print HTML of feature G> if (F is not used) { <print HTML of feature E>} … Performance optimization has to be adaptive! Thousands of dead CSS rules in the package.

17. Adaptive Static Resource Optimization Challenges: • Deep Integration • Viral Adoption • Agile Development Responses: • Build interface to separate requirement declaration and delivery of static resources • Requirement declaration: lives with HTML generation, logged • Delivery: Globally optimized by analyzing the logged trace

18. Adaptive Packager: Designs

19. Interfaces • Back to Day 1: require_static(A_css); <render HTML of feature A> require_static(B_css); <render HTML of feature B> require_static(C_css);<render HTML of feature C> <deliver all required CSS> <print all rendered HTML> Separate Declaration from actual Delivery Requirement Declaration lives with HTML Global Optimization on Delivery

20. Packager: Global JS/CSS Optimization Offline analysis Online process require_static(A_css); <render HTML of feature A> require_static(B_css); <render HTML of feature B> require_static(C_css); <render HTML of feature C> <deliver all required CSS> <print all rendered HTML> Usage Pattern logs Packaging algorithm “Optimal” packages

21. Packager: Usage Pattern Logs Usage Pattern logs

22. Packager: Cost/Benefit Model • To package two files A & B: • “Cost”: for page requests that only uses A, we waste the bytes of B, vice versa • “Benefit”: for page requests that uses both A and B: we save one round trip • Bytes / Bandwidth ~ Latency • “Profit” to be maximized: Benefit – Cost

23. Packager: Cost/Benefit Model • Assume: latency = 40ms, and bandwidth = 1 Mbps • A+B: 40ms * 11.131M • No cost, pure gain. • Definitely package

24. Packager: Cost/Benefit Model • Assume: latency = 40ms, and bandwidth = 1 Mbps • B+C: 40ms * 11.1M • – 300B / 1Mbps * 0.031M • Benefit larger than cost • OK to package

25. Packager: Cost/Benefit Model • Assume: latency = 40ms, and bandwidth = 1 Mbps • B+D: 40ms * 1K • – 2K / 1Mbps * 11.13M • Cost larger than benefit • Don’t package

26. Packager: Algorithms • Greedy algorithm • Simulated Annealing • Clustering algorithms • … Packaging algorithm

27. Packager: Optimal packages • Pkg 1: A, B, C • Pkg 2: E, F, G Usage Pattern logs Packaging algorithm “Optimal” packages

28. Adaptive Static Resource Optimization Adaptive Packaging / Spriting • Cross-feature optimizations (e.g. search + ads) • Adaptive to change of user behaviors and code developments • Same methodology works for image spriting

29. Adaptive Static Resource Optimization Adaptive progressive rendering • flush the commonly used JS/CSS/Images to the browsers, before the page is generated at the server • Browser downloads JS/CSS in parallel with page generation at server side

30. Adaptive Progressive Rendering Adaptive progressive rendering • Progressive rendering: flush the commonly used JS/CSS/Images to the browsers, before the page is generated at the server

31. Adaptive progressive rendering What are the common JS / CSS should we flush? Usage Pattern logs

32. Adaptive progressive rendering Sum on the page name:

33. Adaptive progressive rendering Remove the rare usages

34. Adaptive Static Resource Optimization Adaptive progressive rendering • More than 500 msec gain for home page end-to-end time • Adaptive to the change of popular resources • Need to solve issues with redirections, HTTP-only cookie and buffers

35. Adaptive Packager: Operations

36. Update frequency • Too slow: fail to adapt to usage changes • Too fast: invalidate caches • Usually weekly • Can be smarter

37. Resources for new features • No usage logs available • Developers to estimate the initial usage pattern • Update packages one day after release, if necessary

38. Deployment status since Nov 2008 • Rebalance weekly based on previous week’s usage pattern (>100K unique usage patterns) • Javascript and CSS packaging only (image soon)

39. Scalable with development JS/CSS Code base: Nov 2008 -> Sep 2009

40. Adaptive to new usages # of JS/CSS Packages served in the past month

41. Experiment: Adaptive Image Spriting The puzzle of image spriting: • Thousands of virtual gifts with static images, which to sprite?

42. Experiment: Adaptive Image Spriting The puzzle of image spriting: • The answer is…

43. Lessons learnt

44. Regression detection can be trickier “The page has 5KB more Javascripts!” • Regression introduced by Packager? • Or, by the code?

45. When data go wrong

46. “All models are wrong. Some models are useful.” • Not all users have 1Mbps and 40ms RTT connections • A/B Testing to find an optimal model • Customized packaging for user groups • Per browser groups: ~5% improvements • Locale / IP Prefix might yield better performance

47. Human errors are possible Automatic analysis is preferable: require_static(A_css); //forgot to remove the require_static require_static(B_css); <render HTML of feature B> require_static(C_css); <render HTML of feature C> <deliver all required CSS> <print all rendered HTML>

48. Summary • Performance optimization must be adaptive • Good interface helps optimization • Trace-based analysis is powerful • Always monitor the optimization effect

49. Thank you!