Learn how to manipulate data and create machine learning feature sets in Spark using SQL in Python.

If you’re familiar with SQL and have heard great things about Apache Spark, this course is for you. Apache Spark is a computing framework for processing big data, and Spark SQL is a component of Apache Spark. This four-hour course will show you how to take Spark to a new level of usefulness, using advanced SQL features, such as window functions.



Over the course of four chapters, you’ll use Spark SQL to analyze time series data, extract the most common words from a text document, create feature sets from natural language text, and use them to predict the last word in a sentence using logistic regression.



You’ll start by creating and querying an SQL table in Spark, as well as learning how to use SQL window functions to perform running sums, running differences, and other operations.



Next, you’ll explore how to use the window function in Spark SQL for natural language processing, including using a moving window analysis to find common word sequences.



In chapter 3, you’ll learn how to use the SQL Spark UI to properly cache DataFrames and SQL tables before exploring the best practices for logging in Spark.



Finally, you use all of the skills learned so far to load and tokenize raw text before extracting word sequences. You’ll then use logistic regression to classify the text, using raw natural language data to train a text classifier.



This course provides a thorough introduction to Spark SQL, and by the end, you will have a firm grasp of the basics and will understand how Spark combines the power of distributed computing with the ease of use of Python and SQL.