py4JJavaError：调用 o253.load 时出错。 : java.lang.ClassNotFoundException: 找不到数据源: bigquery

Question

Trying to read data from bigquery to jupyter notebook with pyspark libraries. All of the apache spark and java hvae been downloaded to my C:Drive. Read and watched tutorial videos but none of them which seem to work. looking for guidance

Code:

import pyspark 
import findspark
from pyspark import SparkContext,SparkConf 
from pyspark.sql import SparkSession
from pyspark.sql.functions import window, col, year, month, aggregate, date_add, 
timestamp_seconds, rank, split
from pyspark.sql.types import StructField, StructType, StringType, BooleanType, DoubleType, 
StringType, IntegerType, FloatType
#import com.google.cloud.spark.bigquery 
#this creates spark UI - check current spark session
spark =SparkSession.builder.master('local[*]').appName('conversions').enableHiveSupport().getOrCreate() 
df = spark.read.format('bigquery').load('table')
df.show()

error: Py4JJavaError: An error occurred while calling o253.load. : java.lang.ClassNotFoundException: Failed to find data source: bigquery. Please find packages at http://spark.apache.org/third-party-projects.html

Answer 1

Please change the SparkSession creation to

spark =SparkSession.builder \
  .master('local[*]') \
  .appName('conversions') \
  .enableHiveSupport() \
  .conf('spark.jars.packages', 'com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.23.2') \
  .getOrCreate() 

Also, please make sure you are using a python notebook rather than a pyspark notebook - otherwise Jupyter will create the SparkSession for you and no additional packages can be added.

See more documentation in theconnector's repo .