Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Training | Edureka

1. PySpark Tutorial Copyright © 2018, edureka and/or its affiliates. All rights reserved.

2. Objectives of Today’s Training 1 PySpark 2 Advantages of PySpark 3 PySpark Installation 4 PySpark Fundamentals 5 Demo Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

3. PySpark Copyright © 2018, edureka and/or its affiliates. All rights reserved.

4. Spark Ecosystem MLlib (Machine Learning) Spark Streaming (Streaming) GraphX (Graph Computation) Spark SQL (SQL) Apache Spark Core API Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

5. Python in Spark Ecosystem MLlib (Machine Learning) Spark Streaming (Streaming) GraphX (Graph Computation) Spark SQL (SQL) Apache Spark Core API Python API for Spark(PySpark) Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

6. PySpark Spark is an open-source cluster-computing framework which is built around speed, ease of use, and streaming analytics Python is general purpose high level programming language. It provides wide range of libraries and is majorly used for Machine Learning and Data Science • It is a Python API for Spark majorly used for Data Science and Analysis • Using PySpark, you can work with Spark RDDs in Python Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

7. Advantages Spark with Python Copyright © 2018, edureka and/or its affiliates. All rights reserved.

8. Advantages E A SY TO L E A RN Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

9. Advantages E A SY TO L E A RN SIM PL E & C OM PRE H E N SIVE A PI Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

10. Advantages E A SY TO L E A RN SIM PL E & C OM PRE H E N SIVE A PI B E TTE R C OD E RE A D A B IL ITY & M A IN TE N A N C E Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

11. Advantages E A SY TO L E A RN SIM PL E & C OM PRE H E N SIVE A PI B E TTE R C OD E A VA IL A B ITL ITY OF VISU A L IZA TION RE A D A B IL ITY & M A IN TE N A N C E Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

12. Advantages E A SY TO L E A RN SIM PL E & W ID E RA N G E OF L IB RA RIE S C OM PRE H E N SIVE A PI B E TTE R C OD E A VA IL A B ITL ITY OF VISU A L IZA TION RE A D A B IL ITY & M A IN TE N A N C E Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

13. Advantages E A SY TO L E A RN A C TIVE C OM M U N ITY SIM PL E & W ID E RA N G E OF L IB RA RIE S C OM PRE H E N SIVE A PI B E TTE R C OD E A VA IL A B ITL ITY OF VISU A L IZA TION RE A D A B IL ITY & M A IN TE N A N C E Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

14. PySpark Installation Copyright © 2018, edureka and/or its affiliates. All rights reserved.

15. PySpark Installation 1. Go to: https://spark.apache.org/downloads.html 2. Select the Spark version from the drop down list 3. Click on the link to download the file. Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

16. PySpark Installation Install pip (version 10 or more) Install jupyter notebook Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

17. PySpark Installation Add the Spark and PySpark in the bashrc file Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

18. PySpark Fundamentals Broadcast & Accumulator Spark Context SparkFiles StorageLevel RDDs SparkConf DataFrames MLlib Copyright © 2018, edureka and/or its affiliates. All rights reserved.

19. Broadcast & Accumulator SparkFiles StorageLevel Spark Context RDDs SparkConf DataFrames MLlib

20. Spark Context SparkContext is the entry point to any spark functionality Local Cluster Py Process Worker (JVM) Spark Context Pipe Socket Block 1 Py4J Py Process Pipe Spark Context Py Process Worker(JVM) Pipe Local FS Socket Block 2 Py Process Pipe Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

21. Spark Context SparkContext parameters Master appName sparkHome pyFiles Environment batchSize Serializer conf Gateaway JSC Profiler_cls Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

22. Spark Context SparkContext parameters Master appName sparkHome pyFiles Environment batchSize Serializer conf Gateaway JSC Profiler_cls Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

23. PySpark Basic life cycle of a PySpark program 01 03 Create RDDs Cache RDDs Create RDDs from some external data source or parallelize a collection in your driver program. Cache some of those RDDs for future reuse Lazy 04 Perform Actions Transformation Lazily transform the base RDDs into new RDDs using transformations Perform actions to execute parallel computation and to produce results 02 Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

24. Broadcast & Accumulator Spark Context SparkFiles StorageLevel SparkConf DataFrames MLlib RDDs

25. Resilient Distributed Dataset (RDDs) RDDs is the building block of every Spark application and is immutable R D D esilient Fault tolerant and is capable of rebuilding data on failure istributed Data is distributed among the multiple nodes in a cluster ataset Collection of partitioned data with primitive values or values of value Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

26. Transformations & Actions in RDDs To work on this immutable data, you need to create a new one via Transformations and Actions Transformations Actions ❑ map ❑ collect ❑ flatMap ❑ collectAsMap ❑ filter ❑ reduce ❑ distinct ❑ countByKey/countByValue ❑ reduceByKey ❑ take ❑ mapPartitions ❑ first ❑ sortBy Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

27. Broadcast & Accumulator Spark Context SparkFiles StorageLevel RDDs SparkConf DataFrames MLlib

28. Broadcast & Accumulator Parallel processing is achieved in Spark by using shared variables Shared Variables Broadcast Accumulator These variables are used to aggregate the information through associative and commutative operations These variables are used to save the copy of data across all nodes Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

29. Broadcast & Accumulator Spark Context SparkFiles StorageLevel RDDs DataFrames MLlib SparkConf

30. SparkConf SparkConf provides the configurations to run a Spark application on a local system or a cluster SparkConf object is used to set different parameters which takes priority over the system properties Once SparkConf object is passed to Spark, it becomes immutable Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

31. SparkConf Attributes of SparkConf class set(key, value)……………………………………… set(key, value)……………………………………… Sets Config property Sets the master URL setMaster setMaster(value)…………………………………… (value)…………………………………… Sets an application’s name setAppName setAppName(value)………………………………… (value)………………………………… Gets the configuration value of a key get(key, get(key, defaultValue defaultValue=None)……… =None)……… Sets the Spark installation path on worker nodes setSparkHome setSparkHome(value)…………………………… (value)…………………………… Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

32. Broadcast & Accumulator SparkFiles Spark Context StorageLevel RDDs SparkConf DataFrames MLlib

33. SparkFiles SparkFiles class helps in resolving the paths of files added to the Spark get(filename)…………………………………………… get(filename)…………………………………………… It specifies the path of the file that is added through sc.addFile() It specifies the path to the root directory of the file that is added through sc.addFile() getrootdirectory getrootdirectory()……………………………… ()……………………………… Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

34. Broadcast & Accumulator Spark Context StorageLevel SparkFiles RDDs SparkConf MLlib DataFrames

35. DataFrames Dataframe is a distributed collection of rows under named columns Immutable Lazy Evaluations Distributed Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

36. Dataframes D A TA RDBMS Col 1 Col 2 … Col n Row 1 Row 2 : Row 3 RDDs Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

37. Broadcast & Accumulator StorageLevel Spark Context SparkFiles RDDs SparkConf DataFrames MLlib

38. StorageLevels Class StorageLevel decides how RDDs should be stored Disk Serialize Replicate Memory Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

39. Broadcast & Accumulator Spark Context SparkFiles StorageLevel RDDs SparkConf DataFrames MLlib

40. MLlib Machine Learning API in Spark which interoperates with NumPy in Python is called MLlib It provides an integrated Data Analysis workflow Enhances speed and performance Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

41. MLlib Various algorithms supported by MLlib MLlib Clustering Frequent Pattern Matching Linear Algebra Collaborative Filtering Classification Linear Regression Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

42. MLlib Various algorithms supported by MLlib MLlib Clustering Frequent Pattern Matching Linear Algebra Collaborative Filtering Classification Linear Regression Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

43. MLlib Various algorithms supported by MLlib MLlib Clustering Frequent Pattern Matching Linear Algebra Collaborative Filtering Classification Linear Regression Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

44. MLlib Various algorithms supported by MLlib MLlib Clustering Frequent Pattern Matching Linear Algebra Collaborative Filtering Classification Linear Regression Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

45. MLlib Various algorithms supported by MLlib MLlib Clustering Frequent Pattern Matching Linear Algebra Collaborative Filtering Classification Linear Regression Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

46. MLlib Various algorithms supported by MLlib MLlib Clustering Frequent Pattern Matching Linear Algebra Collaborative Filtering Classification Linear Regression Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

47. MLlib Various algorithms supported by MLlib MLlib Clustering Frequent Pattern Matching Linear Algebra Collaborative Filtering Classification Linear Regression Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

48. MLlib Various algorithms supported by MLlib MLlib Clustering Frequent Pattern Matching Linear Algebra Collaborative Filtering Classification Linear Regression Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

49. Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training