1 / 35

Game Development Telemetry

Game Development Telemetry. Georg Zoeller, BioWare. Georg Zoeller , Lead Technical Designer, BioWare Austin. georg@gulbsoft.org. About. http://www.linkedin.com/in/georgzoeller. Georg Zoeller. 7 player characters with unique content and quests. 4 male, 3 female player appearances.

kellsie
Download Presentation

Game Development Telemetry

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Game Development Telemetry Georg Zoeller, BioWare

  2. Georg Zoeller, Lead Technical Designer, BioWare Austin georg@gulbsoft.org About http://www.linkedin.com/in/georgzoeller Georg Zoeller

  3. 7 playercharacters with unique content and quests. • 4 male, 3 female player appearances. • 10 dedicated story testers. • 9 male and 1 female. Guess which appearances rarely got used? Once upon a time…. …in the ‘Jade Empire’ QA department

  4. Dragon Age: Origins • ~800,000 words of dialog voiced by • 144 voice actors in English alone. • 180+ area layouts. • 3 classes, 3 races, 2 gender options. • 10 companion characters. • 6 unique, hour long Origin Stories. • 300+ abilities • 300,000+ lines of script code. • 3,000 Cutscenes, 5,500 animations, 20,000 models • New IP, World, Rules, Engine. Full Path coverage is hard to achieve, and even harder to verify. QA resources are finite and sometimes volatile. Planning and managing for full path coverage is very time intensive. The Problem.

  5. n.  The science and technology of automatic measurement and transmission of data by wire, radio, or other means from remote sources, as from space vehicles, to receiving stations for recording and analysis. Telemetry - GameDevelopment Telemetry – …automatic measurement and transmission of data from game executable, build pipeline and development tools for recording, analysisand workflow improvement! …not from space, yet Definition.

  6. Customer facing Developer facing • Understand how customers experience the game. • Identify and react to issues post launch. • Understand the market, learn for future games. • Already quite popular. • Understand howdevelopers use tools and pipeline. • Identify and react to pipeline and workflow issues . • Measure stability, productivity and efficiency of development. • Is Business Intelligence. Same idea, different goals.

  7. 2nd Generation homegrown telemetry system. Serving Dragon Age: Origins, Mass Effect2, SWTOR • On Dragon Age: Origins alone: • 1,100,000+ games tracked on 1,141 different machines. • Tracked any game of DA:O in the entire EA network. • 250 GB of data gathered in ~38,000,000 datapoints • 50,000+ bugs, 4597 unique users on 582 machines. Let’s have a look… All your data…. Introducing ‘SkyNet’

  8. Know who is running which build, language, executable. • Drill down into any running game…

  9. Chronological / spatial event visualization.

  10. Aggregated spatial visualization (hmp) Click image to jump to location in game!

  11. Raw data is great… ... if your day starts with SELECT socks, pants FROM closet; • Engineers demand different detail than managers. • Executive summaries are … graphs. • Best solution we found: Drill-down Data Visualization Why presentation makes or breaks your system.

  12. Report view provides the next level of detail. • Aggregates information for leads and intermediate managers. • Reports are nothing more than SQL statements. • But can be exported and subscribed to. • Allow further drilldown…

  13. This view is interesting to programmers, not so much managers. • Incoming callstacks are timestamped, sanitized and hashed. • Tells us the rough# of unique crashes in the system • Freshly introduced crashes file an ‘investigation task’ to engineering QA. • Who can retrieve the full details by drilling down…

  14. Business Intelligence – Great for evaluation outsourcer performance

  15. Now let’s use the telemetry data to improve other workflow... • Industry Typical Bugtracker interface • Lots of fields • Some mandatory • Many freeform. • Result: • Fields stay empty • Random values, Typos • Database pollution. Smart Bugs When bugs learned to listen…

  16. Telemetry enabled interface. • Retrieves fields from the server. • No typos in database fields. • Fewer mandatory fields. • Links to session history, RunInfo, etc. • Location aware bugs. • Suggests who should get the bug. • Warns of duplicates based on text, build and location. • Vacation Tracker Aware. • Smart Categories. • But wait, there is more… Better?.

  17. 95% adoption rate for new interface after 3 months. • Significant Bug count increase (especially non QA). • Significant Bug quality increase. • Bugs always have platform and build #. • Hardware information in each bug. • Much cleaner bug database, spatial bug information. • Spatial information == great for finding dupes. • Increased overall QA capacity by drafting everyone to help. • Much improved reporting… Results.

  18. Achievements work! • They influence player behavior through reward. • Teach about deeper aspects of the game. • Yield valuable telemetry for developers. • Inspire some gamers to go to great length... … and many developers are gamers, so: • Yes, they work for (some) developerstoo. • Can be used to influence how testers play. • Yield valuable balance telemetry for design. • Inspire some developers to play the game. A lot. Achievements For developers?

  19. Doubled hours of playtime for a month after introduction • Increased bugs filed by both QA and engineers. • Got people competing and excited about the game. Morale boosters. • Taught users about more obscure game features. • Some people stayed ‘addicted’ and had to get them all. • Reward desirable behavior… Results.

  20. Telemetry systems don’t mix well! • Different goals, different needs. • Different technical requirements • Start small, don’t over-engineer. • Be Non Invasive. • Reuse test hooks • Make it dead simple to add events. • Create an API and they will come… Getting Started Practical tips and pitfalls.

  21. C# Server • Receives Packets (udp) • Tracks clients. • Mangles packets into DB. DB ASP.NetWebApp SkyNet API Tools and Sensors Our Tech.

  22. UDP vs TCP? • Lightweight, non blocking. • We use one way communication. • Telemetry data may get lost. • Server software • C# is RAD. • Runtime-Compiled ‘packet handlers’ • Dynamically routes events into tables by type. • Protocol • Character delimited string. • Only defined in packet handlers. • No support for fragmentation. More Tech Talk.

  23. Game Client / Server • Callstacks, Warnings, etc. • Game Events, Movement, etc. • Savegames, Screenshots • Performance Timings • Toolset • Startup time, Usage patterns. • Callstacks, Warnings, etc. • Wikis / Sharepoint. • Bugtracker • Landesk • IP enabled coffee maker. • Twitter • Goes well with timeline of PR Useful data sources.

  24. Usage patterns.

  25. One database that indexes information from any available datasource in the company. • Makes it searchable and viewable. • And retains historical information so we can understand, optimize and learn. • Push information back into the game (e.g. warn users when playing a broken build). • Something like this… Endgame Where do we go from here?

  26. Questions, Comments, Feedback? If you have further questions -or end up implementing your own development telemetry system, I’d like to hear from you – especially if you have ideas for additional data sources. Georg Zoeller gdc@gulbsoft.org Thank you. For attending this presentation.

More Related