1 / 73

Remixing web data for your hacks the easy way

Remixing web data for your hacks the easy way. YQL – the can opener of the web. Hacking together systems in 24 hours is a lot of fun. But you want to spend that time thinking about the interface. And not how you get to the right data in the right format.

abril
Download Presentation

Remixing web data for your hacks the easy way

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Remixing web data for your hacks the easy way YQL – the can opener of the web

  2. Hacking together systems in 24 hours is a lot of fun.

  3. But you want to spend that time thinking about the interface.

  4. And not how you get to the right data in the right format.

  5. The web is full of juicy and long lasting data. http://www.flickr.com/photos/clspeace/162336973/

  6. And there is a lot of it around. http://www.flickr.com/photos/tudor/2981410947/

  7. However, our attempts to get to it can be clumsy. http://www.flickr.com/photos/lumachrome/2140368742/

  8. What we need is an easy way to get to that data. http://www.flickr.com/photos/careytilden/115435226/

  9. We had a way to do that for quite a while now.

  10. http://pipes.yahoo.com/

  11. Pipes, however is high end technology… http://www.flickr.com/photos/axio/2346342672/

  12. We’re developers, not interface users. http://www.flickr.com/photos/codepo8/2278641937/

  13. So for a long time people asked Yahoo for a command line version of pipes.Can we have one?

  14. Yes, we can!

  15. The Yahoo Query Language, or short YQL is a unified interface language to the web.http://developer.yahoo.com/yql/

  16. Using YQL, accessing the web and its services becomes as easy as SQL:select {what} from {service} where {condition}

  17. Say you want kittens in your hack (who doesn’t?)select * from flickr.photos.search where text="kitten"

  18. Say you want kittens in your hack (who doesn’t?)

  19. Say you only want 5 kittensselect * from flickr.photos.search where text="kitten" limit 5

  20. Say you only want 5 kittens

  21. Nice, but where can you get this?

  22. YQL is a REST API in itself, and it has two endpoints.

  23. The public endpoint does not need any authentication.http://query.yahooapis.com/v1/public/yql?q={query}&format={format}

  24. The private endpoint needs oauth authentication.http://query.yahooapis.com/v1/yql?q={query}&format={format} http://developer.yahoo.com/yql/guide/authorization-access.html

  25. Output formats are XML or JSON. JSON also allows for a callback parameter to use the output directly as JSON-P.

  26. http://query.yahooapis.com/v1/public/yql?q=select%20*%20from flickr.photos.search%20where%20text=%22kitten%22limit%205&format=json&callback=meow

  27. You can mix and match several web services using the in() command.

  28. Guess what this does:select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)

  29. Find photos in London, UK with a Creative Commons “By” license and give me all the information you have about them.

  30. select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)

  31. Using a command like this and some out-of-the-box UI elements like Yahoo Maps and the YUI carousel, you can build something *very* quickly.

  32. http://isithackday.com/hacks/cantine/index.php?loc=covent+gardenhttp://isithackday.com/hacks/cantine/index.php?loc=covent+garden

  33. Instead of selecting all the information you can also limit the results:select name,url from upcoming.venue where metro_id in (select id from upcoming.metro where search_text="stokey")

  34. select name,url from upcoming.venue where metro_id in (select id from upcoming.metro where search_text="stokey")

  35. The diagnostics part of the returned data shows you what happened and how long it took

  36. Here’s what Yahoo offers you in this format:

  37. Here’s what Yahoo offers you in this format:

  38. You want even more? http://www.flickr.com/photos/verylastexcitingmoment/3123597774/

  39. Alright, how about this? json microformats rss xml atom csv feed html

  40. Telegraph’s headlines anyone? select * from html where url='http://www.telegraph.co.uk/' and xpath='//h3/a'

  41. Telegraph’s headlines anyone?

  42. You can dynamically create YQL queries to collate several sources…

  43. Then use cURL to pull them off the web with one single http request!

  44. YQL caches and compresses the results for you.Pretty easy, isn’t it?

  45. Here’s another fun part:Anyone can be part of the YQL interface.

  46. All we need the data provider to do is to create a schema that explains their data structure.http://developer.yahoo.com/yql/guide/yql-opentables-chapter.html

  47. And people do… http://github.com/spullara/yql-tables/tree/master

  48. For example the national maritime museum: select * from nmm.archive.search where searchterm=‘horatio nelson'

  49. select * from nmm.archive.search where searchterm=‘horatio nelson'

More Related