Learning in the Wild

1. Learning in the Wild In the Meeting Room Scenario Satanjeev “Bano” Banerjee Dialogs on Dialog March 18th, 2005

2. Introduction • Project CALO: Building agents that support professionals in their work life • Smart management of emails, calendar, etc. • Organizing information from meetings for easier/faster access • Inter-disciplinary, distributed research: • Speech, Vision, Multi-modal integration, Machine learning, Reasoning, etc…

3. CMU’s Piece of the Pie • Automatic meeting understanding, specifically: • Perform automatic speech recognition • Detect different meeting states • Presentations, discussions, briefings • Detect major topics of discussion • Detect decisions and action items • Detect personnel skills • Vision expert vs. speech expert vs. statistics expert… • What else? • Program’s main aim: Learning in the Wild!

4. Learning in the Wild! • Improve performance with continued use and without additional engineering! • Typical learning scenario: • Engineers deliver the system to company X, and never touch the system again. • Compare % of action items automatically detected during 1st and 10th weeks of use. • % action items detected during 10th week must be greater than during 1st week!

5. How can we do LITW? • Purely unsupervised learning techniques: • Improve ASR through adaptation • Clustering to create better understanding of topics / personnel skills… • “Implicitly supervised” learning • Learn from users doing the same task (hints) • Ask for “labels” from human in particularly “useful” cases (nudges)

6. Description of Software • Meeting Recorder • Speech, video, notes, slides, whiteboard markings • All synchronized using NTP server, and auto-stored in central database • Mock Brow • Plays back media recorded using Meeting Recorder • Offline “understanding” component • Speech recognizer • Meeting state and topic detector

7. Out-of-the-Box Topic Detection • Look for sudden drastic changes in • Vocabulary • Subset of active speakers • Look for cue phrases, intonation • “So…” • “Moving on…”

8. Can we Improve with Supervision? • If a user was to manually mark topics, can we improve beyond a trained system? • Yes, by acquiring the vocabulary of the various topics typical of this company! • Training data may not cover company’s domain • Yes, by learning the active speakers for each topic • Training data cannot cover this at all • Yes, by learning idiosyncrasies of this group • Topic shifts happen only when Bob is speaking • Bob always says “Now, the next thing…” before he shifts

9. How to Elicit Supervision? • But user will not mark topics for us • Because cost too high, and value too low! • So: Can we design an interaction so topic markup falls out of other user activity? • An activity with “viable” cost/value ratio • Idea: Better designed notes box

10. First allow user to enter the agenda, one item per box User will enter notes on discussion of each agenda item in the appropriate box From times of taking notes we can tell topic boundaries! “Smart” Notes Box Report on integration Inputs from sphinx ok Trouble with output of adapter Discuss web page Web space set up Bob needs access to CGI server Plan roadmap

11. Cost / Value of Smart Notes Box • Cost of interaction: Nearly same as “flat” notes box • Immediate value: • Users can “one click email” the notes to other participants – useful for meeting scribe • Longer term value: • Notes will be “media augmented” – users will be able to use Mock Brow to play back the media, using the notes as an index • At future meetings, MR will “suggest” agenda items – even less typing needed than “flat” notes box!

12. Steps to Create Instances of Learning in the Wild • Identify task to be performed by system • E.g.: Topic detection • Create “out-of-the-box” system • E.g.: Decision trees trained on manual data • Show that supervision can help • Design interface to extract implicit supervision

13. My Research Goals • For the chosen tasks (topic detection, action item detection, etc): • Create out-of-the-box technologies • Implement/develop algorithms to rapidly adapt using sparse implicit supervision • Design interfaces that extract maximal implicit supervision • Wish: Somehow generalize all this…

14. How to Evaluate? • Out-of-the-box technology: Standard machine learning train/test evaluation • Improvement with adaptation: Show delta of performance between system with/without supervision • Evaluating the interface’s cost/value viability: Long and situated user study to show that indeed regular users will use the interface, often!

15. User Study: Testing the Smart Notes Box Interface • Goals: • Will users use the “smart” notes box? • Will their use indeed line up cleanly with actual topic boundaries? • Cannot run a “one off” lab test • Users may use the box the first few times – but will they give up after a few meetings? • Will users use the notes box when they are deeply involved in the meeting?

16. User Study Design • Invite multiple groups to use the Meeting Recorder for their regular meetings • At the first meeting, describe the interface • Point out the value of taking notes in the smart notes box instead of in Emacs say • Observe the groups over a couple of months, noting their use of the text box

17. User Study Success Conditions • Will declare victory if users use the smart text box often • Often defined as some percentage of maximum use we can hope for = we get supervision for all the major topics discussed! • Will pop the champagne if their use of the text box increases over time • May happen as they experience the long term values of the interaction