Evolving Internet Technologies: Web Search Engines

1. Evolving Internet Technologies:Web Search Engines Danny SullivanEditor, SearchEngineWatch.comhttp://searchenginewatch.com/

2. Overview • Key “technology” in 2001 was survival • Crawlers replacing humans • New & old players to watch • 11 September & Mindreading • Other Things

3. RIP 2001 Go.com (Infoseek) You were one of the first web-wide spiders andlater added your own human directory of sites NBCi (Snap)You provided your own human-compiledguide to the web ExciteYou were another of the oldest web spiders to finally cease crawling

4. Search Economics • Economics is boring but important! • Makes search engines viable; may impact results • Banner ads no longer sell • Listing services new way to make money • Allow much needed “conversation” betweensearch engines and site owners… • But more interactive with results than banners,so searchers and site owners have new fears • What’s offered & should you worry?

5. Paid Placement • Buy your way to the top • All sell it, even Google • Overture (GoTo) sells for AOL, AltaVista, Ask, HotBot, Lycos, Yahoo and others • In Europe, Espotting sells for Yahoo, Lycos, others

6. Paid Placement Concerns • Users don’t really seem to mind -- yet • Similarity to “editorial” may cause distrust • Main reason behind FTC complaint last July • Ask Jeeves, Lycos recently improved labels • Why deny users top sites, if they don’t pay??? • Heavy “ad break” might drive users away…

7. Meta Search or Meta Ads? • Meta Ads • Dogpile, http://www.dogpile.com • Search.com, http://www.search.com • Mamma, Metacrawler like above • Meta Search • Vivisimo, http://vivisimo.com • IxQuick, http://www.ixquick.com • qbSearch, http://www.qbsearch.com • SurfWax, http://www.surfwax.com

8. Paid Submission • Pay to get your site reviewed quickly • No guaranteed ranking – no guarantee to even be included! • Yahoo and LookSmart both offer • Mandatory for business categories • Annual charge at Yahoo: Yellow Pages

9. Paid Submission Concerns • Is it fair to miss some businesses? • How many florists do you want? 100, 1000? • What about non-profits, hobbyists? • Non-commercial categories exempt at Yahoo • LookSmart’s use of Zeal.com feeds itsnon-commercial listings, give good balance

10. Paid Inclusion • Get deeper representation in listingsand with crawlers, faster revisits • Usually doesn’t guarantee rankings, but… • Like having more tickets in the lottery – more chances to win • Every major crawler but Google sells this, as does LookSmart

11. Inktomi: $39 gets first URL listed in2 days, revisited each week Want more, $12-15 each, or CPC pricing No pay? Still might get included, anyway Program has provisions for non-profits No rank boost Paid Inclusion: Example

12. Paid Inclusion Concerns • Will we see important sites / pages dropped just because they don’t pay? • That works against users and site owners • Fair those who pay better represented? • The “real” world works this way • Northern Light worked this way for years • May depend on a case-by-case basis

13. Humans Were Supreme • From start of popular use of the web, human-powered Yahoo has been top search site • Why? It helped you refine. Search for “travel” gave 10 categories rather than 10 million results • Yahoo “seemed” to find things when it actually gave you less but forced you to be more specific Others followed Yahoo’s lead…

14. Rise & Fall Of Humans • By 2000, 5 major human directories “powered” 6 of top 10 search engines • But now, 3 directories power 4 of top 10 • Yahoo, MSN, Netscape, LookSmart • AOL, Lycos, Ask Jeeves abandoned humans in 2001/2002 Why the change?

15. Human Weaknesses • Editors cost money • Go and NBCi ran out of this in 2001 • Ask also scaled back on human answers • Machines can now do some of what humans originally did… • “Related Searches” refine queries in the way categories did • “Autocategorization” also refines…

16. Group pages into categories, on the fly One reason why Teoma, Wisenut and Vivisimo get good reviews Not news to Northern Light! Google says not necessary, but we’ll see They find human categorization better (directory tab) Auto-Categorization

17. But Humans Still Involved • Crawlers better at being “human” because they leverage human work more than in the past • Human-made links used to determine importance • Links used to determine context of pages • Links used to autocategorize into “communities” • Crawlers also dependent on directories, giving them great weight in considering how to rank • So what happens now that Yahoo & LookSmart are more commercial? • What happens if the Open Directory fails?

18. Who’s New: Crawlers • Teoma.com • Potential there, but will Ask have the funds and know how (this time) to make it happen • Coverage is set to grow; of course, paid inclusion was first “improvement” shown • WiseNut.com • LookSmart set to buy it; will this solve the freshness issue?

19. Who’s Still Hot • Google (and Google Toolbar) • Everything they do is magic • Good: finally, a tool you can learn and depend on • AllTheWeb.com • Big improvements recently; take another look • Don’t forget to visit Yahoo’s categories or surprise yourself with a search at MSN

20. People hit Google & others for news Terms included: cnn, news, world trade center, bbc, reuters, msnbc, sky news, new york times, pentagon, bin laden, american airlines, united airlines What did they get?… Lessons Of 11 September

21. Google results, 4 hours after attack

22. FAST/AllTheWeb.com results, 4 hours after attack

23. Ask Jeeves results, 4 hours after attack

24. But at AltaVista, less than 2 hours after the attack…

25. Why Did AltaVista Succeed? “Blended” results mixed in news content,even if news option on home pagehad been ignored We know historically that home page optionsDO get ignored, but we learn this again onSept. 11, by watching Google and others

26. Sept. 11 dramatically illustrates the main search challenge – the need to somehow automatically hit the correct dataset Images for search on “pictures of spain” MP3 files for search on “madonna” Movie info for search on Harry Potter this month How NOT to do it, then good examples… Read My Mind

29. Smart query analysis, then suggestionsor insertions of non-web material Products at AltaVista Sidebar results at AllTheWeb News, dictionary, stock & more at Google Encyclopedia at MSN Careful not to take away all control Power search for few who want to drive Examples Of Mind Reading

30. To mind read, you need specialty datasets Among the majors, Google & AllTheWeb pushing here, & I think they’ll keep going Also think (and hope) we’ll still see more “vortals” or “vertical portals” Moreover, http://www.moreover.com MessageKing, http://www.messageking.com xrefer, http://www.xrefer.com LawCrawler, http://www.lawcrawler.com Specialty Search / Vortals

31. Freshness Size “Off the page” ranking criteria Other Issues & Trends

32. AllTheWeb pushed end of last year to be 9-12 days old at most – now more likely to be a month, like others Google & others aiming to be less than a month old or fresher for key documents Just show dates when pages were visited! Feeling For Freshness

33. Still Growing, But Still Missing • The leaders? • Google, 1.1 to 1.6 billion documents • AllTheWeb, 625 million • Large index probably more comprehensive • We do want more index growth! • However, don’t judge a search engine only based on its index size…

34. Does Size Matter? • To professionals, yes. Coverage helps them find unusual or obscure material • What good is half a haystack? • To average users, not really. They desperately need better relevancy. • How about I dump a haystack on your head? • 100 million extra pages makes no difference tobest matches for “horoscopes” or “britney spears”

35. “Off The Page” Ranking • Looking beyond content of the page,since webmasters can’t easily control this • Link analysis still going strong • But can produce oddities, like infamous Bush result • Under new pressure from link spammers • Clickthrough measurements not as hot • Personalization might get revived with Google • Past fears would limit results, rather than help

36. Some Closing Thoughts Search engines are the top resource for Americans seeking answers, used 32% of the time --Consumer Daily Question Study, Fall 2000 • Yet we’ve had them less than 10 years! • Answers to everything weren’t on web before, aren’t now and never will be, so…

37. Just One Of Many Tools • Don’t expect miracles from search engines • They’re great “Swiss Army Knives,”but you’ll still want an entire toolbox • My hot search tools? Telephone & Email! • Use them to avoid “search rage” • Stop searching after 10 minutesand try other means. Also…

38. Be Non-Traditional • Forget Boolean, please • Don’t cast your net wide • You don’t need every synonym in your query… • Instead, explore what’s in the first catch! • Unlike traditional tools, web documents LINK • A few good pages usually lead you to more good pages – your answer may be a few clicks away • You’ll also find links bring you to documents that contain the synonyms you would have tried

39. This Presentation - http://calafia.com/presentations Search Engine Watch - http://searchenginewatch.com Web Searching Tips – Search Engine Listings Free Search Engine Newsletters(SearchDay – Search Engine Report) Become A Member – You Support Me & Chris Sherman!(and get some extra benefits) http://searchenginewatch.com