Reading data from sql server in databricks(pyspark)

Question

I have a task where I need to read data from Sql server table in Databricks notebook. I am using Pyspark for the transformation.

My question is how many connection pyspark will open with sql server to read the data? does it depend on the number of maps created by the job?

Thanks

Answer 1

By default, Spark will read data via JDBC only into one partition, so it should open one or two connections. But this is very inefficient, so it's recommended to specify lowerBound , upperBound , and numPartitions parameters, so reading of data will happen in parallel - but this is heavily dependent on if you have a column (numeric, timestamp or something like) that could be used to split data into given number of partitions. With these options, Spark will open something like min(numPartitions, num CPU cores) connections.