Adding StructType columns to PySpark DataFrames
In this article, we are going to learn about adding StructType columns to Pyspark data frames in Python.
The interface which allows you to write Spark applications using Python APIs is known as Pyspark. While creating the data frame in Pyspark, the user can not only create simple data frames but can also create data frames with StructType columns. This can be easily done by defining the StructType columns in the schema for the data frame and then creating the data frame later on.
What is StructType?
The class that is used to programmatically specify the schema to the data frame and create complex columns like nested struct, array, and map columns is known as StructType. The StructType can be imported through the following command in Python:
from pyspark.sql.types import StructType
The StructType contains a class that is used to define the columns which include column name, column type, nullable column, and metadata is known as StructField.
Stepwise Implementation to add StructType columns to PySpark DataFrames:
Step 1: First of all, we need to import the required libraries, i.e., libraries SparkSession, StructType, StructField, StringType, and IntegerType. The SparkSession library is used to create the session while StructType defines the structure of the data frame and StructField defines the columns of the data frame. The StringType and IntegerType are used to represent String and Integer values for the data frame respectively.
from pyspark.sql.types import StructType, StructField, StringType, IntegerType from pyspark.sql import SparkSession
Step 2: Now, create a spark session using the getOrCreate() function.
spark_session = SparkSession.builder.getOrCreate()
Step 3: Then, define the data set in the list.
data_set = [((nested_values_1), column_value_1), ((nested_values_2), column_value_2), ((nested_values_3), column_value_3)]
Step 4: Moreover, define the structure using StructType and StructField functions respectively.
schema = StructType([StructField('column_1', StructType( [StructField('nested_column_1', column_type(), True), StructField('nested_column_2', column_type(), True), StructField('nested_column_3', column_type(), True) ])), StructField('column_2', column_type(), True)])
Step 5: Further, add StructType columns to the Pyspark data frame, i.e., create a Pyspark data frame using the specified structure and data set.
df = spark_session.createDataFrame(data = data_set, schema = schema)
Step 6: Finally, display the data frame.
df.show()
Example 1:
In this example, we have defined the data structure with StructType which has four StructFields βFull_Nameβ, βDate_Of_Birthβ, βGenderβ, and βFeesβ. The StructType βFull_Nameβ is also further nested and contains three StructFields βFirst_Nameβ, βMiddle_Nameβ, and βLast_Nameβ. We have also defined the data set and then created the Pyspark data frame according to the data structure.
Python3
# Python program to add StructType columns to PySpark DataFrames # Import the libraries SparkSession, StructType, StructField, StringType, IntegerType from pyspark.sql import SparkSession from pyspark.sql.types import StructType, StructField, StringType, IntegerType # Create a spark session using getOrCreate() function spark_session = SparkSession.builder.getOrCreate() # Define the data set data_set = [(( 'Vinayak' ,' ',' Rai'), '2000-21-02' , 'Male' , 13000 ), (( 'Ria' , 'Singh' , 'Rajput' ), '2004-01-06' , 'Female' , 10000 )] # Define the structure for the data frame by adding StructType columns schema = StructType([ StructField( 'Full_Name' , StructType([ StructField( 'First_Name' , StringType(), True ), StructField( 'Middle_Name' , StringType(), True ), StructField( 'Last_Name' , StringType(), True ) ])), StructField( 'Date_Of_Birth' , StringType(), True ), StructField( 'Gender' , StringType(), True ), StructField( 'Fees' , IntegerType(), True ) ]) # Create the Pyspark data frame using createDataFrame function df = spark_session.createDataFrame(data = data_set, schema = schema) # Display the data frame having StructType columns df.show() |
Output:
Example 2:
In this example, we have defined the data structure with StructType which has two StructFields βDate_Of_Birthβ and βAgeβ. The StructType βDate_Of_Birthβ is also further nested and contains three StructFields βYearβ, βMonthβ, and βDateβ. We have also defined the data set and then created the Pyspark data frame according to the data structure.
Python3
# Python program to add StructType columns to PySpark DataFrames # Import the libraries SparkSession, StructType, # StructField, StringType, IntegerType from pyspark.sql.types import StructType, StructField, StringType, IntegerType from pyspark.sql import SparkSession # Create a spark session using getOrCreate() function spark_session = SparkSession.builder.getOrCreate() # Define the data set data_set = [(( 2000 , 2 , 21 ), 18 ), (( 1998 , 4 , 16 ), 24 ), (( 1998 , 1 , 11 ), 18 ), (( 2006 , 3 , 30 ), 16 )] # Define the structure for the data frame by adding StructType columns schema = StructType([ StructField( 'Date_Of_Birth' , StructType([ StructField( 'Year' , IntegerType(), True ), StructField( 'Month' , IntegerType(), True ), StructField( 'Date' , IntegerType(), True ) ])), StructField( 'Age' , IntegerType(), True ) ]) # Create the Pyspark data frame using createDataFrame function df = spark_session.createDataFrame(data = data_set, schema = schema) # Display the data frame having StructType columns df.show() |
Output: