how we can upload parquet file in azure blob storage using spark and scala?

Question

 val input_blob_path = "Enter input azure blob path here"
    val output_blob_path = "Enter output azure blob path here"
    val storageAccountName = "Enter_Your_Storage_Account_Name"
    val storageKeyValue = "Enter_Your_Storage_Account_Key"
    val containerName = "Enter_Your_containerName"

    val sparkSession = SparkSession.builder()
      .master("local")
      .appName("WriteDataToAzureBlob")
      .config(
        s"fs.azure.account.key.${storageAccountName}.blob.core.windows.net",
        storageKeyValue)
      .getOrCreate()

    parqDF.write.parquet("wasbs://${containerName}@{storageAccountName}.blob.core.windows.net/${output_blob_path}")

I am using this code to write in blob storage but it's now working for me.

Answer 1

For the error:

scala module 2.13.2 requires jackson databind version >= 2.12.0 and < 2.13.0 in scala

• Try to remove old dependencies of Jackson and upgrade the plugin or added 2.12.0 series
• All of Jackson packages that Azure SDKs use, you may need to pin each of them to a specific version and their version number set to the same one as of Spark.

OR

• Try to add a different com.fasterxml.jackson.core dependencies version as per error and then you may need to override them explicitly.

   "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.12.5"

See 1. scala - Jackson conflicts in Apache Spark when used with Azure Java SDK - Stack Overflow

2. Maven Repository: azure-core-serializer-json-jackson

Other References:

1. Jackson compatibility issues GitHub
2. scala -incompatible Jackson versions -SO , Scala module 2.12.3 requires Jackson Databind version 3 -SO
3. Manage Scala & Java libraries for Apache Spark - Azure Synapse Analytics | Microsoft Docs