Scenario-Based Requirements for Web Macro Tools

1. Scenario-Based Requirements for Web Macro Tools Christopher Scaffidi, Allen Cypher, Sebastian Elbaum,Andhy Koesnandar, Brad Myers

2. PBE and browser automation • Programming-by-example (PBE) is a success story • Research systems: Eager, Marquise • Systems in widespread use: Microsoft Office macros • Researchers have applied PBE to browser automation • e.g.: Internet Scrapbook, Koala, Miro, Robofox, Turquoise • These tools… 1. Watch the user perform a task 2. Infer the user’s intent 3. Record a web macro representing the intent 4. Allow the user to replay the macro on new data introduction scenarios  requirements Robofox  conclusion

4. PBE web macro tools are not widely adopted • The web context introduces new challenges to PBE: • Web site changes can break macros without warning • Need features for detecting and responding to site change • Office tasks involve not just browsersbut also spreadsheets and other applications • Need integration with other applications • What other unmet requirements are there? introduction scenarios  requirements Robofox  conclusion

5. Missing: a methodical characterization of requirements for web macro tools • Many papers use scenarios to explain and motivate a PBE tool’s features. • Such scenarios… • Are retrospective: The scenarios show the features that actually work, rather than opportunities for future work. • Lack pedigree: Are the scenarios hypothetical or based on a user study? Will the tool features be useful in practice? introduction scenarios  requirements Robofox  conclusion

6. Outline • We have… • Analyzed 10 scenarios where web macros would be desirable • Characterized 25 requirements that tools must supportto be useful for many real-world tasks • Used these requirements as a benchmark to identify opportunities for enhancing a particular tool (Robofox) introduction scenarios  requirements Robofox  conclusion

7. Catalog of scenarios • Using a web site to look up information – 6 scenarios • Currency converter, Package tracker, Per diem lookup, Staff lookup, Stock analysis, Watcher for eBay • Typical users: office workers • Using a web form to order products – 1 scenario • Path to procurement • Typical users: online shoppers • “Scraping” a large part of a site – 3 scenarios • Peoplesoft scraper, Person locator scraper, Scraper for CMS • Typical users: System administrators introduction scenarios  requirements Robofox  conclusion

8. Scenarios came from a variety of sources • Contextual inquiry – 3 scenarios • Currency converter, Per diem lookup, Staff lookup • Co-workers – 2 scenarios • Path to procurement, Peoplesoft scraper • Our own experiences – 2 scenarios • Package tracker, Scraper for CMS • Online sources – 3 scenarios • Person locator scraper, Stock analysis, Watcher for eBay introduction scenarios  requirements Robofox  conclusion

9. What is in a scenario? introduction scenarios  requirements Robofox  conclusion * = required section

10. Requirements: Triggering macros • On-demand execution • Required to support users’ work tasks • Scheduled execution • Especially for scrapers • Event-based triggers • Watcher for eBay executes when user visits Google homepage • Subroutines • Especially for lookup macros introduction scenarios  requirements Robofox  conclusion

11. Requirements: Using objects on pages • Text snippets • Almost all scenarios require reading text • Tabular information - • Almost all scenarios read tabular data • Web form widgets • Read/write textbox, dropdown, checkbox widgets • Other HTML structures • Watcher for eBay introduction scenarios  requirements Robofox  conclusion

12. Requirements: Adapting to site changes • Adaptation to changing page layout • Changes to text location, text color, page structure • Adaptation to changing form fields - • Changes to field names and field values • Adaptation to changing URLs • “This page has moved to X. Please update your links.” introduction scenarios  requirements Robofox  conclusion

13. Requirements: Reading and writing dataoutside pages • Browser APIs • Read/write highlighted text, show popup window • Spreadsheets and other files - • Often one record per spreadsheet row or XML node • Parameters containing user input • Prompting for an input (e.g. password) during playback introduction scenarios  requirements Robofox  conclusion

14. Requirements: Transforming data • Reformat to equivalent value - • For comparing values and for making outputs consistent • Extracting values’ parts • For comparing values • Combining values • Such as arithmetic and string concatenation introduction scenarios  requirements Robofox  conclusion

15. Requirements: Executing control structures • Looping operations - • Especially in scrapers • Conditional operations • Macro behavior depends on data provided by user as well as data returned by web sites introduction scenarios  requirements Robofox  conclusion

16. Requirements: Recovering from failure • Partial restarts • Especially for scrapers • Exception handlers • Integration with assertions introduction scenarios  requirements Robofox  conclusion

17. Requirements: Supporting macro maintenance • User-understandable representation • For maintenance and for sharing macros with other users • Editable macros • Analogous to features supported by textual editors • Features for debugging • Traces, breakpoints, step-by-step execution, etc • Maintenance at runtime • On error, enter maintenance mode introduction scenarios  requirements Robofox  conclusion

18. Example benchmark use: Robofox • Inserts “sanity checks” to detect changing page structure. • Includes heuristics for “refinding” page elements based on color, label, DOM location. • Some changes (e.g.: form field renaming) are not handled automatically. introduction scenarios  requirements Robofox  conclusion

19. Example benchmark use: Robofox • Can read/write spreadsheets, but only if they have a specific layout • Can do a “foreach” loop but not a “while(condition)” loop introduction scenarios  requirements Robofox  conclusion

20. Example benchmark use: Robofox • User-defined exception handlers can fire when assertions fail introduction scenarios  requirements Robofox  conclusion

21. Summary of Robofox limitations • Of 25 requirements… • Supported: 13 • Limited support: 5 • Unsupported: 7 • Of 10 scenarios… • Supported: 3 • Limited support: 2 (unsupported scenario variations) • Unsupported: 5 introduction scenarios  requirements Robofox  conclusion

22. Opportunities to enhance Robofox • Adding support for 2 requirements… • Tabular information • Reformat to equivalent value • … would leave only… • 2 unsupported scenarios • 2 unsupported scenario variations introduction scenarios  requirements Robofox  conclusion

23. Extending the benchmark • These 25 requirements are “Version 1” of a benchmark for evaluating web macro tools. • As researchers meet existing requirements, new scenarios and new requirements will appear. • We created a wiki where YOU can upload scenarios. • http://softwaresurvey.cs.cmu.edu/wmcorpus.html introduction scenarios  requirements Robofox  conclusion

24. Thank You… • To VL/HCC for the opportunity to present. • To EUSES Consortium members for suggestions. • To NSF and EUSES for funding(CCF-0325273, CCF-0438929, CCF-0324861). introduction scenarios  requirements Robofox  conclusion