How to use join In Python
We can filter the data with aggregate operations using leftsemi join, This join will return the left matching data from dataframe1 with the aggregate operation
Syntax: dataframe.join(dataframe.groupBy(‘column_name_group’).agg(f.max(‘column_name’).alias(‘new_column_name’)),on=’FEE’,how=’leftsemi’)
Example: Filter data with a maximum fee from all departments
Python3
# importing module import pyspark # importing sparksession from pyspark.sql module from pyspark.sql import SparkSession #import functions from pyspark.sql import functions as f # import window module from pyspark.sql import Window # creating sparksession and giving an app name spark = SparkSession.builder.appName('sparkdf').getOrCreate() # list of student data data = [["1", "sravan", "IT", 45000], ["2", "ojaswi", "CS", 85000], ["3", "rohith", "CS", 41000], ["4", "sridevi", "IT", 56000], ["5", "bobby", "ECE", 45000], ["6", "gayatri", "ECE", 49000], ["7", "gnanesh", "CS", 45000], ["8", "bhanu", "Mech", 21000] ] # specify column names columns = ['ID', 'NAME', 'DEPT', 'FEE'] # creating a dataframe from the lists of data dataframe = spark.createDataFrame(data, columns) # display dataframe.join(dataframe.groupBy('DEPT').agg( f.max('FEE').alias('FEE')), on='FEE', how='leftsemi').show() |
Output:
GroupBy and filter data in PySpark
In this article, we will Group and filter the data in PySpark using Python.
Let’s create the dataframe for demonstration:
Python3
# importing module import pyspark # importing sparksession from pyspark.sql module from pyspark.sql import SparkSession # creating sparksession and giving an app name spark = SparkSession.builder.appName('sparkdf').getOrCreate() # list of student data data = [["1", "sravan", "IT", 45000], ["2", "ojaswi", "CS", 85000], ["3", "rohith", "CS", 41000], ["4", "sridevi", "IT", 56000], ["5", "bobby", "ECE", 45000], ["6", "gayatri", "ECE", 49000], ["7", "gnanesh", "CS", 45000], ["8", "bhanu", "Mech", 21000] ] # specify column names columns = ['ID', 'NAME', 'DEPT', 'FEE'] # creating a dataframe from the lists of data dataframe = spark.createDataFrame(data, columns) # display dataframe.show() |
Output:
In PySpark, groupBy() is used to collect the identical data into groups on the PySpark DataFrame and perform aggregate functions on the grouped data. We have to use any one of the functions with groupby while using the method
Syntax: dataframe.groupBy(‘column_name_group’).aggregate_operation(‘column_name’)
Filter the data means removing some data based on the condition. In PySpark we can do filtering by using filter() and where() function