如何使用 pyspark 从 databricks 集群连接到 mongodb Atlas

Question

how to connect to mongodb Atlas from databricks cluster using pyspark

This is my simple code in notebook

from pyspark.sql import SparkSession
spark = SparkSession \
    .builder \
    .appName("myApp") \
    .config("spark.mongodb.input.uri", "mongodb+srv://admin:<password>@mongocluster.fxilr.mongodb.net/TestDatabase.Events") \
    .getOrCreate()

df = spark.read.format("mongo").load()
df.printSchema()

But I am getting error as

IllegalArgumentException: Missing database name. Set via the 'spark.mongodb.input.uri' or 'spark.mongodb.input.database' property

What is wrong am i doing

Answer 1

I followed this steps and I was able to connect.

• Install org.mongodb.spark:mongo-spark-connector_2.12:3.0.2 maven library to your cluster as I was using scala2.12

• Goto Cluster detail page and in Advance option under Spark tab, you add below two config parameters

   spark.mongodb.output.uri connection-string spark.mongodb.input.uri connection-string

Note connection-string should look like this - (have your appropriate user, password and database names)

mongodb+srv://user:password@cluster1.s5tuva0.mongodb.net/my_database?retryWrites=true&w=majority

• Use following python code in your notebook and it should load your sample collection to a dataframe

# Reading from MongoDB
df = spark.read\
.format("com.mongodb.spark.sql.DefaultSource")\
.option("uri", "mongodb+srv://user:password@cluster1.s5tuva0.mongodb.net/database?retryWrites=true&w=majority")\
.option("database", "my_database")\
.option("collection", "my_collection")\
.load()

• You can use following to write to mongo db

events_df.write\
    .format('com.mongodb.spark.sql.DefaultSource')\
    .mode("append")\
    .option( "uri", "mongodb+srv://user:password@cluster1.s5tuva0.mongodb.net/my_database.my_collection?retryWrites=true&w=majority") \
    .save()

Hope this should work for you. Please do let others know if it worked.