Py4JJavaError: An error occurred while calling o771.save. Azure Synapse Analytics Notebook

Question

Here is my pyspark code used in Notebook

data_lake_container = 'abfss://abc.dfs.core.windows.net' 
    stage_folder = 'abc' 
    delta_lake_folder = 'abc' 
    source_folder = 'abc' 
    source_wildcard = 'abc.parquet' 
    key_column = 'Id' 
    key_column1 = 'LastModifiedDate' 
    
    source_path = data_lake_container + '/' + stage_folder + '/' + source_folder + '/' + source_wildcard 
    
    delta_table_path = data_lake_container + '/' + delta_lake_folder + '/' + source_folder
     
    sdf = spark.read.format('parquet').option("recursiveFileLookup", "true").load(source_path)
     
    if (DeltaTable.isDeltaTable(spark, delta_table_path)):
    
        delta_table = DeltaTable.forPath(spark, delta_table_path)
    
        delta_table.alias("existing").merge(
            source=sdf.alias("updates"),
            condition=("existing." + key_column + " = updates." + key_column + " and existing." + key_column1 + " = updates." + key_column1) # We look for matches on the name column
        ).whenMatchedUpdateAll(
        ).whenNotMatchedInsertAll(
        ).execute()
     
     else:
        sdf.write.format('delta').save(delta_table_path)
    

while executing above code I'm getting below error

Py4JJavaError: An error occurred while calling o771.save.
 : org.apache.spark.SparkException: Job aborted.
 at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231)
 at org.apache.spark.sql.delta.files.TransactionalWrite.$anonfun$writeFiles$1(TransactionalWrite.scala:216)
 at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:107)

Kindly help me in resolving error

Answer 1

Py4JJavaError: An error occurred while calling o771.save.
 : org.apache.spark.SparkException: Job aborted.

The above error generally occurred because of non-compatible versions of spark connector and spark.

Refer - org.apache.spark.SparkException: Job aborted due to stage failure: Task from application

If the above solution does not work for you, please share a full stack trace of error. It is difficult to identify issues with shared information.

Answer 2

@AbhishekKhandave, when I looked into full error there was date column with Data range less than '1900-01-01'. that was the issue finally was abled to run script. thank you for your response