1 / 35

Intelligent Web Agent

Intelligent Web Agent. Under the guidance of Dr. S. B. Nair Presented by Sandeep Kumar (001127) Dept. of CSE, IIT Guwahati. Goals .

winda
Download Presentation

Intelligent Web Agent

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Intelligent Web Agent Under the guidance of Dr. S. B. Nair Presented by Sandeep Kumar (001127) Dept. of CSE, IIT Guwahati

  2. Goals • To develop an Intelligent web agent which learns about the user’s behavior with any user intervention and uses the information gathered to help him search pages of his interests by ranking results of google according to his interests.

  3. Intelligent Web Agent • Agent – something that perceives its environment through its sensors and acting upon it through its effectors. • Web Agent – the environment is the world wide web. • Intelligent Web Agent – A rational web agent i.e. one that can make a rational decision when given a choice leading to its goal. • ‘Personalized’ Intelligent Web Agent (PIWA) – It learns user preferences and behavior over a length of time and exhibits ample intelligence in its decisions.

  4. Problem • Explosive growth of internet has made it the largest knowledge repository mankind ever had. • Need to use it efficiently – providing one with information one seeks. • Searching for information is a Problem. • The internet structure - a massive mess of hyperlinks pointing to HTML pages makes it difficult • Search Engines though excellent in their working fail to satisfy an user’s needs returning millions of results - most of which are unwanted

  5. How can we help the user • A web based search engine lacks information about the user. • They are based on algorithms which need more information about the search query – a normal user provides just one or two. • We can have a software agent at the user’s desktop. • It has more access to user’s browsing behavior. • It can learn from it. • Using its knowledge base, it can help user to find information matching his interests.

  6. Profiling • Static – profiles are built beforehand like templates • Dynamic – profile are generated dynamically learning from user’s behavior. • Need • an intelligent search engine over an general purpose search engine. • Perform real time adaptive learning from monitoring user’s habits with no relevance feedback from user. • Must change according to interest changes in the user

  7. The Algorithm • Salient Features: • Representation of user’s interest as a group of words. • Generation of the group of words by unobtrusive monitoring of user’s browsing habits with no relevance feedback from user. • Dynamic membership of the group of words representing a particular user’s interest. • Using the group of words to improvise web query generation. • Using the group of words to ranks results from general purpose search engine.

  8. Interest • Basic knowledge block of user profile. • Represented by a group of 10 words each having an associated weight and a timestamp. • Weight represents the importance of word in that particular interest. • Eg: • said 16, yesterday 15, city 15, sadr 14, iraq 12, holy 12, news 12, east 11, talks11 , tension 11 • user’s interest in what sadr said yesterday in a holy city of iraq and his talks about tension and east

  9. Generation of Interest • Key Point – unobtrusive, user friendly

  10. Implementation • Agent act as a proxy for user’s browser. • Passive monitoring of incoming traffic. • The 10 words are extracted from the very pages the user browses through. • Extract top words from a HTML document. • Get the page. • Do feature extraction. • Do stop-word removal. • Do stemming.

  11. Generation of Interest • From HTML pages browsed by the user. • Feature extraction done using latest features of HTMLEditorKit (available in JDK SE > 1.4) • HTML tags given weights like title 10, meta-names 6, block-quote 4, boldfaced and underline 2, fontsizes, etc • Content tag given weight 1 (similar to Term Frequency (tf)) • Weights are summed up for all words. • Commonly used words removed by stop words elimination and removing words of length <= 2 • Top 10 words selected. • Morphological analysis not done as many words don’t occur in dictionary like yahoo, and the process still is not very efficient.

  12. Creation of Profile • Get 10 keywords from each page visited • 2 possible cases: • Current page (keywords) matches/is similar to a past interest -> list of interest updated • Current page (keywords) is new -> new interest created • Match if 3 words or more (>= 30%) match between keywords of current page and past interest. • Interest Update:- Sum up the weights of the matched words and get the top 10 from the merged list.

  13. Maintenance of Profile • An optimum size needed as too big will have erroneous interests and has performance problems, a small list may not cover all his interests – at present its at 20 interest. • When an interest is created or updated its timestamp is updated (associated with the 11 word marker “1234567890”) • The product of timestamp and sum of weights of the interest is used to determine which interest will remain in list and which one removed.

  14. Use of Profile for web searches • Direct Searching: user provides search query • 3 cases: • Query matches one interest • Query matches more than one interest • Query does not match any interest • No match: simple google search • More than one match: sum up the words of query in the matched interest and select the one for which sum in maximum. • So now we have one matched interest

  15. Trigger Pair Model • Trigger Pair – get some words from the matched interest which have weights less than the smallest weight for any word of the query. • Done to prevent overshadowing of original query by more popular/weighted words. • 1 word added for single worded query, 2 words for double or more worded query -> to prevent overshadowing. • Trigger Pair refines results from google to a great extent.

  16. Ranking of results to user’s interest • Get top 20 results from Google. • Get top 10 keywords for each of the result • Score each page by summing the product of weights of common words between matched interest and keywords. • Get the top 10 pages based on the score. – 1st Result • Take a arithmetic mean of ranks of Google and rank of the algorithm and get top 10 pages – 2nd Result

  17. Agent Architecture

  18. Implementation Summary • Keyword: String word, Int val • Interest: 10 keywords + 1 marker with timestamp • Profile: 20 interests • Google element: String title, String snippet, String url

  19. Results : 1. IRAQ • 10 pages from news on Iraq dated April 14th 2004. • 7 interests formed, 3 ages merged.

  20. Interests

  21. Points to note • 1st interest: lexington and concorde came in as they were advertisements on the first page – parser is not designed to ignore and considers part of the page • Can be done if structure of page known beforehand but impossible in present case. • Timestamps of 2nd and 10th page merging as both from reuters

  22. Search: IRAQ • Search query: iraq • Matched interest: last one as weight of iraq is maximum in it – 42 • Trigger Pair causes Fallujah to be appended.

  23. Comparision

  24. Points to note • 1st result of Google describes about fallujah and has less elements of fighting in it – hence ranks a poor 10th in the PIWA rank. • 7th result of Google which is basically a discussion board on iraq war with lots of discussion on “iraq, fallujah, marines, coalition, Baghdad” (word in matched interest) ranks first in PIWA rankings. • Mixed Results: Its rank 1 are 2nd of Google and 5th of PIWA.

  25. 2. Sandeep • 3 pages: 2 homepages and 1 resume • 3 interests formed

  26. Search: Sandeep • Search query: sandeep • Kumar appended

  27. Points to note • Matched interest was derived from 10th result of Google but ranked 2nd in PIWA rankings and hence 5th on mixed results. • A page visited in past affects the results greatly. • Sandeep is very general term so results still not much inclined in my favour.

  28. Search: Sandeep Kumar • Appended by “2004 iitg”

  29. 3. India Pakistan Series • 5 news pages dated 14th April 2004

  30. Search: India Pakistan • Appended by “test series”

  31. Points to note • Normal search in google without trigger pair resulted in results on war – not wanted by user • Google's 7th, 8th, 9th and 10th result don’t make it to top 10 of PIWA – shows ranking differences based on user’s interests.

  32. Mixed Results • In classical AI search terminology • Google:- explore strategy – get new results • PIWA:- exploit strategy – use past information to decide new ranks • Mixed:- a 50-50 mix of both, can be changed to explore more initially and then exploit more as in any other learning process

  33. Conclusion • A profile for an user was generated with absolutely no user relevance feedback • Dynamic profile maintenance – continuously updated by new information. • Profile used to improve user’s web searches to suit his interests.

  34. Future Work • Improve GUI: specially for search utility • Support for plugins: to handle non-HTML documents • Support of encoded pages: SSL, gzipped, etc • News reader can be made easily with improved parser knowing the structure of news pages beforehand.

  35. Thank you

More Related