1 / 27

Chris Olston Benjamin Reed Utkarsh Srivastava Ravi Kumar Andrew Tomkins

Pig Latin: A Not-So-Foreign Language For Data Processing. Chris Olston Benjamin Reed Utkarsh Srivastava Ravi Kumar Andrew Tomkins. Research. Data Processing Renaissance. Internet companies swimming in data E.g. TBs/day at Yahoo!

lotte
Download Presentation

Chris Olston Benjamin Reed Utkarsh Srivastava Ravi Kumar Andrew Tomkins

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Pig Latin: A Not-So-Foreign Language For Data Processing Chris Olston Benjamin Reed Utkarsh Srivastava Ravi Kumar Andrew Tomkins Research

  2. Data Processing Renaissance • Internet companies swimming in data • E.g. TBs/day at Yahoo! • Data analysis is “inner loop” of product innovation • Data analysts are skilled programmers

  3. Data Warehousing …? Often not scalable enough Scale • Prohibitively expensive at web scale • Up to $200K/TB $ $ $ $ • Little control over execution method • Query optimization is hard • Parallel environment • Little or no statistics • Lots of UDFs SQL

  4. New Systems For Data Analysis • Map-Reduce • Apache Hadoop • Dryad . . .

  5. Map-Reduce Input records Output records map reduce map reduce Just a group-by-aggregate?

  6. The Map-Reduce Appeal • Scalable due to simpler design • Only parallelizable operations • No transactions Scale $ Runs on cheap commodity hardware SQL Procedural Control- a processing “pipe”

  7. Disadvantages M R 1. Extremely rigid data flow Other flows constantly hacked in M M M R Join, Union Chains Split • 2. Common operations must be coded by hand • Join, filter, projection, aggregates, sorting, distinct • 3. Semantics hidden inside map-reduce functions • Difficult to maintain, extend, and optimize

  8. Pros And Cons Need a high-level, general data flow language

  9. Enter Pig Latin Need a high-level, general data flow language Pig Latin

  10. Outline • Map-Reduce and the need for Pig Latin • Pig Latin example • Salient features • Implementation

  11. Example Data Analysis Task Find the top 10 most visited pages in each category Visits Url Info

  12. Data Flow Load Visits Group by url Foreach url generate count Load Url Info Join on url Group by category Foreach category generate top10 urls

  13. In Pig Latin visits = load‘/data/visits’ as (user, url, time); gVisits = group visits by url; visitCounts = foreach gVisits generate url, count(visits); urlInfo = load‘/data/urlInfo’ as (url, category, pRank); visitCounts = join visitCounts by url, urlInfo by url; gCategories = group visitCounts by category; topUrls = foreach gCategories generate top(visitCounts,10); store topUrls into ‘/data/topUrls’;

  14. Outline • Map-Reduce and the need for Pig Latin • Pig Latin example • Salient features • Implementation

  15. Step-by-step Procedural Control Target users are entrenched procedural programmers The step-by-step method of creating a program in Pig is much cleaner and simpler to use than the single block method of SQL. It is easier to keep track of what your variables are, and where you are in the process of analyzing your data. Jasmine Novak Engineer, Yahoo! With the various interleaved clauses in SQL, it is difficult to know what is actually happening sequentially. With Pig, the data nesting and the temporary tables get abstracted away. Pig has fewer primitives than SQL does, but it’s more powerful. David Ciemiewicz Search Excellence, Yahoo! • Automatic query optimization is hard • Pig Latin does not preclude optimization

  16. Quick Start and Interoperability visits = load‘/data/visits’ as (user, url, time); gVisits = group visits by url; visitCounts = foreach gVisits generate url, count(urlVisits); urlInfo = load‘/data/urlInfo’ as (url, category, pRank); visitCounts = join visitCounts by url, urlInfo by url; gCategories = group visitCounts by category; topUrls = foreach gCategories generate top(visitCounts,10); store topUrls into ‘/data/topUrls’; Operates directly over files

  17. Quick Start and Interoperability visits = load‘/data/visits’ as (user, url, time); gVisits = group visits by url; visitCounts = foreach gVisits generate url, count(urlVisits); urlInfo = load‘/data/urlInfo’ as (url, category, pRank); visitCounts = join visitCounts by url, urlInfo by url; gCategories = group visitCounts by category; topUrls = foreach gCategories generate top(visitCounts,10); store topUrls into ‘/data/topUrls’; Schemas optional; Can be assigned dynamically

  18. User-Code as a First-Class Citizen • User-defined functions (UDFs) can be used in every construct • Load, Store • Group, Filter, Foreach visits = load‘/data/visits’ as (user, url, time); gVisits = group visits by url; visitCounts = foreach gVisits generate url, count(urlVisits); urlInfo = load‘/data/urlInfo’ as (url, category, pRank); visitCounts = join visitCounts by url, urlInfo by url; gCategories = group visitCounts by category; topUrls = foreach gCategories generate top(visitCounts,10); store topUrls into ‘/data/topUrls’;

  19. Nested Data Model • Pig Latin has a fully-nestable data model with: • Atomic values, tuples, bags (lists), and maps • More natural to programmers than flat tuples • Avoids expensive joins • See paper finance yahoo , email news

  20. Outline • Map-Reduce and the need for Pig Latin • Pig Latin example • Novel features • Implementation

  21. Implementation SQL user automatic rewrite + optimize Pig Pig is open-source. http://incubator.apache.org/pig or or Hadoop Map-Reduce cluster

  22. Compilation into Map-Reduce Every group or join operation forms a map-reduce boundary Map1 Load Visits Group by url Reduce1 Map2 Foreach url generate count Load Url Info Join on url Reduce2 Map3 Group by category Other operations pipelined into map and reduce phases Reduce3 Foreach category generate top10(urls)

  23. Usage • First production release about a year ago • 150+ early adopters within Yahoo! • Over 25% of the Yahoo! map-reduce user base

  24. Related Work • Sawzall • Data processing language on top of map-reduce • Rigid structure of filtering followed by aggregation • DryadLINQ • SQL-like language on top of Dryad • Nested data models • Object-oriented databases

  25. Future Work • Optional “safe” query optimizer • Performs only high-confidence rewrites • User interface • Boxes and arrows UI • Promote collaboration, sharing code fragments and UDFs • Tight integration with a scripting language • Use loops, conditionals of host language

  26. Credits Shubham Chopra Alan Gates Shravan Narayanamurthy Olga Natkovich Arun Murthy Pi Song Santhosh Srinivasan Amir Youssefi

  27. Summary • Big demand for parallel data processing • Emerging tools that do not look like SQL DBMS • Programmers like dataflow pipes over static files • Hence the excitement about Map-Reduce • But, Map-Reduce is too low-level and rigid Pig Latin Sweet spot between map-reduce and SQL

More Related