How to loop dataframe column in databricks using pyspark
Question
i want to apply lemmatization for dataframe column using pyspark running in databricks.Refer the images for error.
1 answers
solution1
0 2020-07-17 12:15:48
Import the function lemmatize_sentence() in side the function and then create the UDF that should work. you are getting this error because the import is on driver node not on the entire cluster. when you import this inside the function so in that case while creating the UDF it will send the copy of function to all the nodes in the cluster.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Databricks Pyspark + How to insert a dataframe schema as a column in a dataframe
How to Use DataFrame Created in Scala in Databricks' PySpark
How to unzip a column in a Spark DataFrame using pyspark
How to fetch all rows data from spark dataframe to a file using pyspark in databricks
Perform a user defined function on a column of a large pyspark dataframe based on some columns of another pyspark dataframe on databricks
Upsert in databricks using pyspark
PySpark Replace Characters using regex and remove column on Databricks
How to split dataframe column in PySpark
How to loop nested directories in DataBricks (Python)Pyspark faster and efficient way
How do I add a new column to a Spark DataFrame (using PySpark)?
Related Tags