Pyspark 将 dataframe 写入 bigquery [错误 gs]

Question

I'm trying to write a dataframe to a bigquery table. I have set the sparkSession with the required parameters. However, at the moment of doing the write I get an error:

Py4JJavaError: An error occurred while calling o114.save.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "gs"
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)

The code is the following one:

import findspark
findspark.init()

import pyspark
from pyspark.sql import SparkSession

spark2 = SparkSession.builder\
    .config("spark.jars", "/Users/xyz/Downloads/gcs-connector-hadoop2-latest.jar") \
    .config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.18.0")\
    .config("google.cloud.auth.service.account.json.keyfile", "/Users/xyz/Downloads/MyProject-cd7627f8ef9b.json") \
    .getOrCreate()

spark2.conf.set("parentProject", "xyz")

data=spark2.createDataFrame(
    [
        ("AAA", 51), 
        ("BBB", 23),
    ],
    ['codiPuntSuministre', 'valor'] 
)

spark2.conf.set("temporaryGcsBucket","bqconsumptions")

data.write.format('bigquery') \
    .option("credentialsFile", "/Users/xyz/Downloads/MyProject-xyz.json")\
    .option('table', 'consumptions.c1') \
    .mode('append') \
    .save()

df=spark2.read.format("bigquery").option("credentialsFile", "/Users/xyz/Downloads/MyProject-xyz.json")\
    .load("consumptions.c1")

I don't get any error if taking out the write from the code, so the error comes when trying to write and may be with something related to the auxiliar bucket to operate with bigquery

Answer 1

the error here suggests that it is not able to recognize the filesystem, you can use the below link for adding the support for gs filesystem, it happens because when you write to bigquery the files are loaded to google cloud storage bucket temporarily and then it is loaded into the bigquery table.

spark._jsc.hadoopConfiguration().set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")