How to use list comprehension In Python

Q: How to use list comprehension In Python

A particular way of iterating over a sequence, i.e., a list, a tuple, a dictionary, a set, or a string) is known as for loop. In this method, we will import the CSV file or create the dataset and then apply a transformation using for loop to the multiple columns of the uploaded or the created data frame.

Method 2: Using for loop

A shorter way of creating a new list based on the values of an existing list is known as list comprehension. In this method, we will import the CSV file or create the dataset and then apply a transformation using list comprehension to the multiple columns of the uploaded or the created data frame.

Stepwise Implementation:

Step 1: First, import the required libraries, i.e. SparkSession, col, and upper. The SparkSession library is used to create the session. The col is used to get the column name, while the upper is used to convert the text to upper case. Instead of upper, you can use any other function too that you want to apply on each row of the data frame.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, upper

Step 2: Now, create a spark session using the getOrCreate function.

spark_session = SparkSession.builder.getOrCreate()

Step 3: Then, read the CSV file or create the data frame using the createDataFrame function.

data_frame=csv_file = spark_session.read.csv('#Path of CSV file',
                                              sep = ',', inferSchema = True, header = True)

data_frame=spark_session.createDataFrame([(column_1_data), (column_2_data), (column_3_data)],
                                         ['column_name_1', 'column_name_2', 'column_name_3'])

Step 4: Next, create a list comprehension to traverse all the elements and convert it to uppercase.

updated_data_frame = data_frame.select(
                      *[upper(col(col_name)).name(col_name) for col_name in data_frame.columns])

Step 5: Finally, display the updated data frame in the previous step.

updated_data_frame.show()

Example:

In this example, we have uploaded the CSV file (link), i.e., basically a data set of 5*5 as follows:

Then, we used the list comprehension to apply a transformation to multiple columns ‘name‘ and ‘subject‘ of the Pyspark data frame uppercase through the function upper.

Python3

# Python program to apply a transformation to multiple  
# columns of PySpark dataframe using list comprehension 
  
# Import the SparkSession, col and upper libraries 
from pyspark.sql import SparkSession 
from pyspark.sql.functions import col, upper 
  
# Create a spark session using getOrCreate() function 
spark_session = SparkSession.builder.getOrCreate() 
  
# Read the CSV file 
data_frame=csv_file = spark_session.read.csv( 
      '/content/student_data.csv', 
      sep = ',', inferSchema = True, 
      header = True) 
  
# Apply a transformation to multiple columns of  
# PySpark dataframe using list comprehension 
updated_data_frame = data_frame.select( 
 *[upper(col(col_name)).name(col_name) for col_name in data_frame.columns]) 
  
# Show the updated data frame 
updated_data_frame.show()

Output:

Apply a transformation to multiple columns PySpark dataframe

In this article, we are going to learn how to apply a transformation to multiple columns in a data frame using Pyspark in Python.

The API which was introduced to support Spark and Python language and has features of Scikit-learn and Pandas libraries of Python is known as Pyspark. While using Pyspark, you might have felt the need to apply the same function whether it is uppercase, lowercase, subtract, add, etc. to apply to multiple columns. This is possible in Pyspark in not only one way but numerous ways. In this article, we will discuss all the ways to apply a transformation to multiple columns of the PySpark data frame.