Metrics for MMP Development and Operations Lessons Learned: The Sims Online

1. Metrics forMMP Development and OperationsLessons Learned: The Sims Online Larry Mellon GDC, Spring 2004

2. Metrics: Catch-22 Useful Careful… GI / GO Hard Data Optimization Tool Expensive

3. Importance of Metrics is Relative Mark Twain Lord Kelvin Measure Everything Measure “Just Enough”

4. MMP Complexity Scale Precision Invest No More Than You Need Kelvin Twain $$$ $

5. MMP: Measurement Focal Points Operational Costs Infrastructure Player Actions Economy

6. Similar Use Case: Casinos(Harrah’s “Total Reward”) Unified Player Action DB Casino Casino Casino Table / Machine Table / Machine Table / Machine Track every Player Action Player Player Player

7. Highly Profitable, Highly Popular Analyze: Unified Player Action DB Profit (per Casino, per Player) Patterns of Play Modify: Casino Operations Player Awards Program “This is one of the best investments that we have ever made as a corporation and will prove to forge key new business strategies and opportunities in the future.“ John Boushy (Harrah's CIO, 2000)

8. Designers Engineers Community Managers Operations Server Reactions Player Actions & Persistent Data Metrics System TSO: Live Monitors, Summary Views Embedded Profiler (Server Side) Automated Report Generators

9. Outline Background: Metrics & MMP Implementation Overview Metrics in TSO Applications & Sample Charts Wrapup Lessons Learned Conclusions Questions

10. Implementation:Driving Requirements Low overhead Common Infrastructure Ease of use

11. User User User Process Process Process esperProbe esperProbe esperProbe esperLog esperLog esperLog Esper Architecture Live Server CPUs

12. esperView esperView esperView esperLog esperLog esperLog t1 t2 t3 t4 t5 Event-level Sampling, Aggregated Reporting esperProbes esperStore Min, Max, Av, Count DBImporter esperFetch

13. Esper Probes • Self-organizing: “class” hierarchy • Data driven: new probes and/or new game content immediately visible on web • Example: ESPER_PROBE • (“Object.interaction.%s”, chair->picked) • (“Object.interaction.puppet.%s”, self->picked) • Human-readable intermediate files

14. EsperView: Web-Driven Presentation Daily Reports Report Generator Graph Caching & Archiving Filtering & Meta Data

15. EsperView: Hierarchical Presentation Process-Level Collection Server Cluster Process Class Process Instance

16. Outline Background: Metrics & MMP Implementation Overview Metrics in TSO Applications & Sample Charts Wrapup Lessons Learned Conclusions Questions

17. Applications of Metrics Load Testing (Realistic Inputs) Beta Testing & Live Operations (Game Tuning, Community Management) Load Testing & Live Operations (Server Performance)

18. Live Beta Testers AlphaVilleServers Test Servers Load Testing:“Monkey See / Monkey Do” Sim Actions (Player Controlled) Sim Actions (Script Controlled)

19. Applications of Metrics Load Testing: Realistic Inputs Beta Testing & Live Operations: Tuning/Mngmnt Load Testing & Live Operations: Server Performance

20. History Make Friends, Shake Hands beats out Give Money / Get Money Least Used: DiscoDancing Meta Data

21. Top: TD Dance, Woohoo Bottom: Dance

22. Players per Lot 0 to 70 players: < 2 / lot 70 to 400 players: > 3 / lot

23. Top: Metrics Bug (sorta) Next: Garden Gnome, Toilet Bottom: Buffet Table

24. Most have one Roommate • Hard-core players: 8 Beta: numPlayers by numRMates

25. Highest • Using Skill Objects • Using Pizza Maker • Selling Objects Lowest • Visitor Bonus • New Avatar • Object “Rentals”

26. Using Objects = $$$ • Visitor Bonus == $ Economy: Detailed View

27. Most getting no V. Bonus • Hard-core players: $$ Visitor Bonus: Who Makes Money?

28. 4 of top 5: windows??

29. Not a well-used feature • Shopping least of all House Categories (Beta Test)

30. Community Management Community Actions & Trends Influencing Player Activity Free Content Tracking Problem Players

31. Marketing In-Game Brand Exposure Special Events Press Release Teasers

32. NYEve: Kiss Count Esper Cities     All Cities (extrapolated)===========================================New Year's Kiss      32,560             271,333Be Kissed Hotly        7,674             63,950Be Kissed              5,658             47,150Be Kissed Sweetly     2,967             24,725Blow a Kiss            1,639             13,658Be Kissed Hello        1,161              9,675Have Hand Kissed        415              3,458===========================================Total              52,074433,949

33. Applications of Metrics Load Testing: Realistic Inputs Beta Testing & Live Operations: Tuning/Mngmnt Load Testing & Live Operations: Server Performance

35. DB byte count oscillates out of control

36. A single DB Request is clearly at fault

38. Most-Used DB Queries (unfiltered) 11,000,000 level Queries need attention, and drown out others

39. DB Queries (Filtered) Filters on 11,000,000 level Queries show patterns of 7,000 level Queries

40. Incoming & Outgoing Packets

41. Outline Background: Metrics & MMP Implementation Overview Metrics in TSO Applications & Sample Charts Wrapup Lessons Learned Conclusions Questions

42. Lessons Learned • Implement early • Ownership, senior engineers • Aggregated probes vs event-level tracking • Automation: collect / summarize / alarms • “There can be only one”…

43. Conclusion: Very Useful! Game Design Data Mining On Players: Untapped Gold… Realistic Load Testing Engine Fixes, Optimization Server Cost, Launch Timing Critical Feature: Accessibility

44. Questions Slides available @ www.maggotranch.com/MMP