Why PySpark code hangs for a while and then terminates abruptly while accessing a dataframe

Question

Problem Statement: The PySpark program hangs when it is reading records from a dataframe based on a condition where a particular field is NOT NULL. This field is a string field and may or many not contain a string value in it. Any operation on this string field, such as checking for NULL, calculation of length of the field, will result in hanging the code and then terminating.

Description: For example, in our case, the PySpark program reads the data from a file and loads into the dataframe. A new column named 'ErrorDescription' is added to the dataframe. This field is dynamically populated with the comma separated error descriptions when the data validation for the record fails for any field. At the end of all the checks, when the dataframe is read to identify the records where the 'ErrorDescription' field is is NULL (valid records) then sometimes the activity completes successfully and sometimes the program hangs and then terminates.

What we did so far: We tried to solve this problem by defining the initial value for 'ErrorDescription' as '' or "" instead of 'NULL'. However after processing the records through all the data validation, whenever we checked the dataframe for 'ErrorDescription' to be '' or "" or NULL the process hung and terminated. The confusing part was that, the records were processed through multiple iterations and for initial 2 iterations this check for 'ErrorDescription' worked fine but then for the next iteration it would hung and then terminate. We modified the code to skip this iteration and continue with the next iteration. Again the code successfully completed the first two iteration, skipped the third iteration, successfully executed the fourth iteration and again hung in the fifth iteration and terminated. The behavior of the code was completely irrational. To add to the confusion, the error dataframe was created by checking for the error records in the parent dataframe by checking the ErrorDescription as NOT NULL. However the code was hanging at the stage where the error dataframe was used to load the data into the database. We initially thought it might be database level issue, but eventually found that it was due to lazy evaluation in pyspark that the error dataframe was executed only when it was accessed for loading into the database table.

Answer 1

Solution: To solve this issue, we defined an integer column called 'ErrorFound' along with the column 'ErrorDescription' as given below.

.withColumn("ErrorDescription", lit(""))
.withColumn("ErrorFound",lit(0))

We changed the field Error_Found to 1 whenever there is an error found in the record and continued population of the ErrorDescription column with relevant error description values. Then during the identification of the error records in the dataframe, we checked the 'ErrorFound == 1' instead of checking for ErrorDescription not equal to NULL or '' or "". With this approach all the iteration executed successfully and the code hanging issue was resolved.

Conclusion: So in summary, if your pyspark code is checking for string field for NULL value and the code is hanging and terminating, switch to using the interger value instead if possible. This will solve the issue.