Creating Intelligent Bots

1. Creating Intelligent Bots This Photo by Unknown Author is licensed under CC BY-SA

2. Zoiner Tejada zoinertejada@solliance.net @zoinertejada Solliance Founder, CEO Author of three books on Azure Dual Microsoft MVP– Microsoft Azure & Data Platform Microsoft Regional Director

3. Overview • Background • Cortana & Bots • Services to Add Intelligence

4. en.wikipedia.org - Text under CC-BY-SA license What’s a bot? • A chatbot is a computer program which conducts a conversation via auditory or textual methods. • Such programs are often designed to convincingly simulate how a human would behave as a conversational partner, thereby passing the Turing test. • Chatbots are typically used in dialog systems for various practical purposes including customer service or information acquisition. • Some chatterbots use sophisticated natural language processing systems, but many simpler systems scan for keywords within the input, then pull a reply with the most matching keywords, or the most similar wording pattern, from a database.

5. Let’s Meet Some Bots

6. https://en.wikipedia.org/wiki/Xiaoice - Text under CC-BY-SA license Xiaoice Xiaoice Over 20 million registered users The average user interacts with the service 60 times a month 850 thousand followers on Weibo Ranked as Weibo’s top influencer Rinna 2.2 million followers on Line 34.3 thousand followers on Twitter

7. Eugene Goostman Alan Turing predicted in his 1950 paper Computing Machinery and Intelligence, that by the year 2000, machines would be capable of fooling 30% of human judges after five minutes of questioning. In 2014, Eugene passed that test convincing 33% of the judges.

8. Mitsuku Her intelligence includes the ability to reason with specific objects. Three-time Loebner Prize winnerin 2013, 2016 and 2017

9. Chatbot creation

10. What’s a bot (revisited)? • Bot’s are conversational user interfaces. • They happen with text or with voice. • They make it the job of the computer to figure out you.

11. Cortana • Cortana is a digital assistant that keeps users informed and productive, helping them get things done across devices and platforms

12. Cortana Skills • Skills define the tasks that Cortana can accomplish. • Cortana invokes the skills based on spoken input from the user. • You can extend Cortana by adding your own skills that let your users use Cortana to interact with your service. To extend Cortana, use the Cortana Skills Kit.

13. Cortana Skills Kit

14. Cortana Skills Dashboard • The dashboard lists the skills that you have configured a Cortana channel for.  • Use the dashboard to deploy your skill to yourself (default), a group of users, or the world.  • Publishing your bot in Bot Framework does not make it available to users.  • To make your skill available to users, you must use the dashboard.  • You also use the dashboard to turn on debugging so you can see the message exchange between your bot and Cortana. https://developer.microsoft.com/en-us/cortana/dashboard#!/home

15. Creating a Cortana Skill • Creating a Cortana skill using Bot Framework requires very little Cortana-specific knowledge and primarily consists of building a bot.  • Cortana has both a visual and an audio component.  • For the visual component, Cortana provides an area of her canvas where your bot can display content such as cards.  • For the audio component, your bot includes text or speech synthesis markup language (SSML) in its response. • Cortana reads the text or SSML to the user, thus giving your bot a voice.

16. Running a Skill Locally in Cortana • Register your bot to get your app ID and password. • Connect your skill to the Bot Framework's Cortana channel.

17. Best Practices • Because Cortana is available on many different devices, some may have a screen while others may be a standalone speaker.  • You should ensure that your bot is capable of handling both scenarios.  • The bot framework provides device information as an entity of the activity, which you can use to tailor the interaction.

18. Invoking Skills • Users don't install or activate skills. • Users invoke skills by speaking an invocation phrase to Cortana.  • The invocation phrase includes an invocation name, which uniquely identifies the skill to invoke.  • The invocation name is an essential part of the natural language phrase or utterance that invokes your Cortana skill.

19. Supported Platforms • Cortana skills run any platform that Cortana runs on.  • Cortana is available on • Windows • Android (via Cortana Client app) • iOS (via Cortana Client app) platforms

20. DEMO Cortana Skill

21. OK, So you Know you need a Bot

22. Bot Framework • A skill is a speech bot.  • Use the Bot Framework to build your speech bot, register it, and define your Cortana channel.  • The framework provides a .NET SDK and Node.js SDK that you use to build your bot.  • The Bot Framework SDKs provide features such as dialogs and built-in prompts that make interacting with users much simpler.  • The Bot Framework itself does not provide real-time chat infrastructure (e.g., Skype or Slack provide that).

23. Anatomy of a Bot

24. Key Components in THE Bot Framework • Bot Builder SDK • Azure Bot Service • Bot Framework Portal • Bot Framework Channel Emulator • Bot Connector • Channels • Activities & Messaging • Rich Cards • Dialogs

25. Bot Builder SDK • An SDK for constructing bots that can handle both free-form interactions and more guided conversations where the user selects from possible values • The SDK provides libraries, samples, and tools to help you build and debug bots.  • The SDK contains built-in dialogs to handle user interactions ranging from a basic Yes/No to complex disambiguation. Built-in recognizers and event handlers help guide the user through a conversation. • Available for C# and Node.js

26. Azure Bot Service • The Azure Bot Service provides an integrated environment purpose-built for bot development. • You can write a bot, connect, test, deploy, and manage it from your web browser with no separate editor or source control required.  • Powered by Microsoft Bot Framework and Azure Functions • Provides five bot templates you can choose from when you create a bot.

27. Azure Bot Service Templates

28. Bot Framework Portal • The portal where you register your bot and connect to channels. https://dev.botframework.com/bots

29. Bot Framework Channel Emulator • A desktop application that allows developers to test and debug their bot • Enables inspection of JSON message exchange & error information • Use to test local or remote bots (via an ngrok tunnel)

30. Bot Connector • Connects a bot to one or more channels and handles the message exchange between bot and channel. • Manages “state” of conversation.

31. Channels • The connection between the Bot Framework and communication apps such as Skype, Slack, or your own custom applications (using Direct Line), etc.

32. Activities & Messaging • Activity • Any communication going back and forth across a connector is an activity of some type. • Message • The most common type of activity. • Can be a text string, contain attachments, interactive elements, and rich cards.

33. Rich Cards • Comprises a title, description, link, buttons, and graphics • Multiple rich cards, displayed in either list format or carousel format • Cards for animations, audio, video, hero, thumbnails, receipt and sign-in • Designed to work well across channels https://github.com/Microsoft/BotBuilder-Samples/tree/master/Node/cards-RichCards

34. Dialogs (the logic) • Dialogs help organize the logic in your bot and manage conversation flow. • Dialogs are arranged in a stack, and the top dialog in the stack processes all incoming messages until it is closed or a different dialog is invoked.

35. Dialogs are to bots, like screens are to apps Application Bot Main Screen Root Dialog Search Screen Search Results Screen Search Dialog Search Results Dialog

36. DEMO Coin Toss Bot

37. SO, how can you add intelligence?

38. How can we do Machine Learning on Azure? SQL Server MachineLearning Services CognitiveServices R Server onHDInsight Azure ML

39. Cognitive Services Web API’s Vision From faces to feelings, allow your apps to understand images and video Language Process text and learn how to recognize what users want • Emotion API • Face API • Computer Vision API’s (analyze image content, detect celebrities, smart thumbnails, OCR) • Text Analytics (sentiment analysis, key phrase extraction, language detection) • Web Language Model API (word break, text conditional and join probabilities, completions) • Language Understanding Intelligent Service API (contextual models) Speech Hear and speak to your users by filtering noise, identifying speakers, and understanding intent Recommendations Tap into rich knowledge amassed from the web, academia, or your own data • Speaker Recognition API • Bing Speech API (speech to text, intent recognition, text to speech) • Recommendations API (Frequently Bought Together, Item to Item Recommendation, Customer to Item Recommendation)

40. Face & Emotions API • Detect the range of feelings on people in a photo

41. Face API • Verify faces are similar

42. Face API • Tag identified faces in images

43. Face API • Find similar looking faces. input results

44. Face API • Group faces by similarity

45. Computer Vision API • Analyze image contents

46. Computer Vision API • Read printed text in images

47. Computer Vision API • Read hand written text in images

48. Computer Vision API • Recognize celebrities and landmarks in images

49. Computer Vision API • Create thumbnails intelligently

50. QnA Maker