1 / 23

CSC 221: Introduction to Programming Fall 2013

Big data building lists list comprehensions, throwaway comprehensions conditional comprehensions processing large data files example: dictionary, anagram finder example: NBA stats, player reports example: twitter stats, hashtag reports read-store-and-process vs. read-and-process.

chung
Download Presentation

CSC 221: Introduction to Programming Fall 2013

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Big data building lists list comprehensions, throwaway comprehensions conditional comprehensions processing large data files example: dictionary, anagram finder example: NBA stats, player reports example: twitter stats, hashtag reports read-store-and-process vs. read-and-process CSC 221: Introduction to ProgrammingFall 2013

  2. Review: building strings • we have seen numerous examples of building strings

  3. Similarly: building lists • since lists & strings are both sequences, can similarly build lists

  4. List comprehensions • Python has an alternative mechanism for building lists: comprehensions • inspired by math notation for defining sets • squares(N) = • in Python:[EXPR_ON_VAR for VAR in LIST]

  5. More comprehensions…

  6. Even more…

  7. Exercises: • use list comprehensions to build the following lists of numbers: • a list of the first 20 odd numbers • a list of the first 20 cubes • a list of the first 20 powers of 2 • given a list of words, use list comprehensions to build: • a list containing all of the same words, but capitalized • a list containing all of the same words, but each word reversed • a list containing the lengths of all of the same words

  8. Throwaway comprehensions • comprehensions can be useful as part of bigger tasks

  9. Another example note: the string method join appends the contents of a list into a single string, using the specified divider

  10. Conditional comprehensions • comprehensions can include conditions • in general: [EXPR_ON_VAR for VAR in LIST if CONDITION]

  11. Another example

  12. Big data • lists enable us to store and process large amounts of data • dictionary.txt contains 117,633 words • suppose we want to write a function that finds anagrams • repeatedly prompt the user for a word • traverse the dictionary to find & display all anagrams of that word • we could start fresh for each word – read from the dictionary file and compare words • however, 117,633 words don't take up that much space (!) • can read and store them in a list, then repeatedly access – MUCH FASTER • how can we determine if two words are anagrams? • generate the alphagram of each word: • "spear"  "aeprs" "pares"  "aeprs • if the alphagrams are identical, the words are anagrams

  13. Anagram finder

  14. NBA stats • NBA2013.txt contains NBA stats from the 2012-2013 season • downloaded from NBA.com (61 KB) • 469 rows (excluding the header) • if a player played for multiple teams in the year (e.g., as a result of a trade), there is a separate entry for each team • each row consists of 29 data fields • line[0] = firstName line[1] = lastName line[2] = (team) line[27] = PPG

  15. Stats list • recall that infile.read() reads the entire contents of infile into a single string • repeatedly searching the string for player info would be tedious • better to break the string into logical divisions, i.e., a list of player data sets [['FIRST', 'LAST', 'TEAM', 'GP', 'MIN', 'W', 'L', 'Win%', 'FGM', 'FGA', 'FG%', '3FGM', '3FGA', '3FG%', 'FTM', 'FTA', 'FT%', 'OREB', 'DREB', 'REB', 'AST', 'TOV', 'STL', 'BLK', 'PF', 'DD2', 'TD3', 'PTS', '+/-'], ['A.J.', 'Price', '(WAS)', '57', '22.4', '21', '36', '0.368', '2.8', '7.2', '39.0%', '1.2', '3.5', '35.0%', '0.9', '1.1', '79.0%', '0.4', '1.6', '2.0', '3.6', '1.1', '0.6', '0.1', '1.3', '1', '0', '7.7', '-1.1'], ['Aaron', 'Brooks', '(HOU)', '53', '18.8', '20', '33', '0.377', '2.7', '6.0', '45.3%', '0.9', '2.5', '37.3%', '0.8', '1.0', '76.9%', '0.2', '1.3', '1.5', '2.2', '1.3', '0.6', '0.2', '1.8', '0', '0', '7.1', '-3.8'], ['Aaron', 'Gray', '(TOR)', '42', '12.2', '14', '28', '0.333', '1.1', '2.1', '53.3%', '0.0', '0.0', '0.0%', '0.5', '1.0', '52.3%', '1.1', '2.0', '3.2', '0.8', '0.9', '0.2', '0.1', '2.0', '1', '0', '2.8', '-2.1'], . . . ] QUESTION: should we store the header in the list?

  16. Processing the stats • for each player name entered by the user: • traverse the playerStats list, looking at each player data set • if the first & last names match, then print the name, team & PPG

  17. Processing the stats use a comprehension to build a list of [PPG, PLAYER] pairs & sort then, display the highest ranking players from the end of the sorted list (be careful that there are that many players)

  18. Really big data • if the data can be stored in a list, it can be accessed faster and repeatedly • sometimes, there is just too much data to store • infochimps.com provides twitter data sets for research & analysis • collected from March 2006 to November 2009 • > 35 million users, > 500 million tweets • there are 41,746,479 lines in the raw data file (1.81 GB) • each line consists of 4 fields, identifying tokens taken from tweets • tokenType either hashtag, url, or smiley • yearAndMonth e.g., 200901for January, 2009 • count # of times that token appeared during that month • token the text of the token WAY TOO BIG TO FIT IN A PYTHON LIST!!!

  19. Preprocessing • searching the entire data set takes too long • better to partition it into more manageable subsets • we only care about hashtags, so filter out the other entries • also, we will search based on year, so can group data into separate files by year since the filtered files only contain hashtags, can omit the first field use to generate hashtags2006.tsv, …, hashtags2009.tsv

  20. Mining the data • the yearly hashtag files are small enough to store and process • hashtags2006.tsv  18 lines (???) • hashtags2007.tsv  2,821 lines • hashtags2008.tsv  98,261lines • hashtags2009.tsv  1,588.960 lines [ … ['200810', '1', 'binary'], ['200812', '2', 'binary'], ['200810', '1', 'binauralbeats'], ['200812', '1', 'binbegeistert'], ['200811', '1', 'bindabei'], ['200808', '1', 'binford'], ['200805', '1', 'bingemans'], ['200810', '5', 'bingo'], … ]

  21. Tag volume • suppose we want the tag volume per month and year • for each month, • use a comprehension to build a list of the counts from entries that corresponds to that month • call the sum function on that list to get the monthly volume • also add the monthly count to a running total for the year

  22. Most frequent tag • can look at historical snapshots by finding the most popular tags • for each month, identify those entries corresponding to that month • use a comprehension to construct a list of (count,tag) pairs • then use max to find the pair with largest count What is [0, "NONE"] for?

  23. Most frequent tag for the year • unfortunately, identifying the most frequent tag for the year is trickier • need to have a separate pass through the data, • must identify and combine multiple entries with same tag

More Related