Categories
28°C, New York
Home Python How to usemap() in Python

How to usemap() in Python

Method 5: Using list comprehension

In this method, we will use map() function, which returns a new vfrom a given dataframe or RDD.  The map() function is used with the lambda function to iterate through each row of the pyspark Dataframe.

For looping through each row using map() first we have to convert the PySpark dataframe into RDD because map() is performed on RDD’s only, so first convert into RDD it then use map() in which, lambda function for iterating through each row and stores the new RDD in some variable then convert back that new RDD into Dataframe using toDF() by passing schema into it.

Syntax:

rdd=dataframe.rdd.map(lambda loop: (
      loop["column1"],...,loop["columnn"]) )
 rdd.toDF(["column1",.......,"columnn"]).collect()

Example: Here we are going to iterate ID and NAME column

Python3




# importing module
import pyspark
 
# importing sparksession from pyspark.sql module
from pyspark.sql import SparkSession
 
# creating sparksession and giving an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()
 
# list  of employee data
data = [["1", "sravan", "company 1"],
        ["2", "ojaswi", "company 1"],
        ["3", "rohith", "company 2"],
        ["4", "sridevi", "company 1"],
        ["5", "bobby", "company 1"]]
 
# specify column names
columns = ['ID', 'NAME', 'Company']
 
# creating a dataframe from the lists of data
dataframe = spark.createDataFrame(data, columns)
 
# select id and name column using map()
rdd = dataframe.rdd.map(lambda loop: (
    loop["ID"], loop["NAME"]))
 
# convert to dataframe and display
rdd.toDF(["ID", "NAME"]).collect()
 






                        









													
													
												

												
												

													
													
Output:

													
													
												

												
												

													
													
[Row(ID='1', NAME='sravan'),
Row(ID='2', NAME='ojaswi'),
Row(ID='3', NAME='rohith'),
Row(ID='4', NAME='sridevi'),
Row(ID='5', NAME='bobby')]

													
													
												

												
												

													
													

													
													
												

												
												

													
													

													
													
												

												
												

													
													

													
													
												

												
											

											
											

												
How to Iterate over rows and columns in PySpark dataframe

												
												

													
													
In this article, we will discuss how to iterate rows and columns in PySpark dataframe.

													
													
												

												
												

													
													
Create the dataframe for demonstration:

													
													
												

												
												

													
													




Python3















                                    




                                    

                                    




                                    




                                    

                                    

                                    
                                




















# importing module


import pyspark


 


# importing sparksession from pyspark.sql module


from pyspark.sql import SparkSession


 


# creating sparksession and giving an app name


spark = SparkSession.builder.appName('sparkdf').getOrCreate()


 


# list  of employee data


data = [["1", "sravan", "company 1"],


        ["2", "ojaswi", "company 1"],


        ["3", "rohith", "company 2"],


        ["4", "sridevi", "company 1"],


        ["5", "bobby", "company 1"]]


 


# specify column names


columns = ['ID', 'NAME', 'Company']


 


# creating a dataframe from the lists of data


dataframe = spark.createDataFrame(data, columns)


 


dataframe.show()



















                        







                        









													
													
												

												
												

													
													
Output:

													
													
												

												
												

													
													

													
													
												

												
											

										


										
										

											
Similar Reads

											
											

												
													

														Method 1: Using collect()

													

														...
													

												
											

											
											

												
													

														Method 2: Using toLocalIterator()

													

														This method will collect all the rows and columns of the dataframe and then loop through it using for loop. Here an iterator is used to iterate over a loop from the collected elements using the collect() method....
													

												
											

											
											

												
													

														Method 3: Using iterrows()

													

														...
													

												
											

											
											

												
													

														Method 4: Using select()

													

														It will return the iterator that contains all rows and columns in RDD. It is similar to the collect() method, But it is in rdd format, so it is available inside the rdd method. We can use the toLocalIterator() with rdd like:...
													

												
											

											
											

												
													

														Method 5: Using list comprehension

													

														...
													

												
											

											
											

												
													

														Method 6: Using map()

													

														This will iterate rows. Before that, we have to convert our PySpark dataframe into Pandas dataframe using toPandas() method. This method is used to iterate row by row in the dataframe....
													

												
											

											
										


									

									

										
Tags:

										
										#Python-Pyspark
										
										#Python
										
										#python
										

									

									

										

											
											
												
												Method 5: Using list comprehension