1 / 21

Large-Scale Collection of Application Usage Data to Inform Software Development

This paper discusses the integration of large-scale application usage data and user feedback to inform software development. It aims to identify and resolve mismatches between user expectations and actual usage, thereby improving the design-use fit. Through techniques such as Expectation-Driven Event Monitoring (EDEM), practitioners can deploy agents that unobtrusively gather data and feedback, even as software is actively used. This innovative approach allows developers to adapt applications based on real user interactions, enhancing the overall quality and relevance of software solutions.

saskia
Download Presentation

Large-Scale Collection of Application Usage Data to Inform Software Development

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Large-Scale Collection of ApplicationUsage Data to Inform Software Development David M. HilbertDavid F. Redmiles Information and Computer Science University of California, Irvine Irvine, California 92697-3425 {dhilbert,redmiles}@ics.uci.edu http://www.ics.uci.edu/pub/eden/

  2. Background Design Design Support - Argo Usage Data and Feedback Collection Support - EDEM Incorporate results ofreview into design Deploy software withexpectation agents Review Use Group Memory and Awareness - Knowledge Depot Usage Data and Feedback Collection Support - EDEM Report results of usage dataand feedback collection

  3. Motivation • Expectations influence designs, designs embody expectations • Mismatches between expectations and how applications are actually used can lead to breakdowns. • Identification and resolution of mismatches can help improve fit between design and use • Identifying mismatches entails observing actual use and comparing it against expectations. • Our research: techniques to enable large-scale incorporation of usage data and user feedback in development to help uncover mismatches and improve the design-use fit.

  4. The Internet • On the positive side • cheap, rapid, large-scale distribution of software for evaluation • simple transport mechanism for usage information and feedback • use and development increasingly becoming concurrent • should make incorporating usage data and user feedback easier • On the negative side • reduces opportunities for traditional user testing • increases variety and distribution of users and usage situations • lack of scalable tools and techniques for incorporating usage information and feedback on a large scale

  5. Problems • Current approaches suffer from significant limitations • usability testing => scale (size, scope, location, duration) • beta testing => data quality (incentives, knowledge, detail) • monitoring => scale & data quality (abstraction, selection, reduction, context, evolution) • The subjective feedback paradox • users not having problems provide feedback => problems • users having problems don’t provide feedback => no problems • The impact assessment and resource allocation problem • what is the impact of suspected problems and proposed solutions? • where should development and evaluation effort be focused?

  6. Approach • Developers • design applications and identify usage expectations • create agents to collect usage data and user feedback • Agents • deployed over the Internet to run on user computers (via HTTP) • perform abstraction, selection, reduction, context-capture as needed to allow actual use to be compared against expectations • report data and feedback to developers (via E-mail) • Data and feedback • inform further evolution of expectations, application, and agents

  7. Approach • Expectation-Driven Event Monitoring (EDEM)

  8. Usage Scenario (Expectations) • A cargo query form

  9. Usage Scenario (Usage Data) • Agents monitor use and collect data unobtrusively • Agents may post messages (optional)

  10. Usage Scenario (User Feedback) • Users may learn more about expectations (optional) • Users may provide feedback (optional)

  11. Usage Scenario (Review) • Data and feedback stored for later analysis and review

  12. Agent Authoring

  13. Agent Specs saved w/ URL Development Computer DevelopmentComputer Java Virtual Machine Java Virtual Machine AgentSpecs CollectedData Top Level Window& UI Events Top Level Window& UI Events ApplicationUI Components ApplicationUI Components EDEMActive Agents EDEMActive Agents Property Queries Property Queries HTTPServer EDEMServer Property Values Property Values User Computer Agent Reports sent via E-mail Agent Specs loaded via URL Architecture

  14. Reference Architecture DataCapture Abstraction, Selection, Context, Reduction DataPackaging DataTransport DataPrep DataAnalysis SystemModel ofUI & App: Components Events Properties Methods AnalystModel ofUI & App: Features, Dialogs, Controls, User-Supplied Values, User Tasks Mapping

  15. Reference Architecture (Word IV) Instrumentation intertwined w/ app DataCapture Abstraction, Selection, Context, Reduction DataPackaging DataTransport DataPrep DataAnalysis SystemModel ofUI & App: Components Events Properties Methods AnalystModel ofUI & App: Features, Dialogs, Controls, User-Supplied Values, User Tasks Mapping

  16. Reference Architecture (Office IV) Event monitoring infrastructure DataCapture Abstraction, Selection, Context, Reduction DataPackaging DataTransport DataPrep DataAnalysis TestWizard Database of Office UI SystemModel ofUI & App: Components Events Properties Methods AnalystModel ofUI & App: Features, Dialogs, Controls, User-Supplied Values, User Tasks Mapping

  17. Reference Architecture (EDEM) Event monitoring infrastructure DataCapture Abstraction, Selection, Context, Reduction DataPackaging DataTransport DataPrep DataAnalysis “Pluggable” Data Abstraction, Selection, Context-Capture, and Reduction Expectation Agents SystemModel ofUI & App: Components Events Properties Methods AnalystModel ofUI & App: Features, Dialogs, Controls, User-Supplied Values, User Tasks Mapping

  18. Conclusions • Usage expectations • help guide data collection • raise awareness of implications of design decisions • Agent architecture • abstraction, selection, reduction in-context and prior to reporting • independent evolution of instrumentation and application • Combined • higher quality data (v. beta tests) with less restrictions on evaluation size, scope, location, duration (v. usability tests)

  19. Possible Applications • Use of long-term data about user and users’ behavior to support • adaptive UI and application behavior • “smarter” delivery of help/suggestions/assistance • Support for monitoring of component-based systems in which • event and state information can be easily “tapped” • low-level data must be related to higher level concepts of interest • available information exceeds that which can practically be collected • data collection needs evolve over time more quickly than application

  20. Future Research • Large-scale evaluation of research in practice • maintenance issues, how is data used, cost-benefit analysis • Relationships to other requirements-related artifacts • e.g., use cases, cognitive walkthroughs, task analysis • Other types of analysis • changes in usage over time • usage involving multiple cooperating users • Reuse and adaptability of infrastructure • generalize to support monitoring of arbitrary software components

  21. For More Info • http://www.ics.uci.edu/pub/eden/

More Related