spark-cassandra (java.lang.NoClassDefFoundError: org/apache/spark/sql/cassandra/package)

Question

i am trying to read DataFrame from cassandra 4.0.3 with spark 3.2.1 using scala 2.12.5 and sbt 1.6.2 but i have a problem.

this is my sbt file:

name := "StreamHandler"

version := "1.6.2"

scalaVersion := "2.12.15"

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "3.2.1" % "provided",
    "org.apache.spark" %% "spark-sql" % "3.2.1" % "provided",
    "org.apache.cassandra" % "cassandra-all" % "4.0.3" % "test",
    "org.apache.spark" %% "spark-streaming" % "3.2.1" % "provided",
    "com.datastax.spark" %% "spark-cassandra-connector" % "3.2.0",
    "com.datastax.cassandra" % "cassandra-driver-core" % "4.0.0" 
)
libraryDependencies += "com.datastax.dse" % "dse-java-driver-core" % "2.1.1" % "provided" 
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "3.2.1" % "provided"
libraryDependencies += "org.apache.commons" % "commons-math3" % "3.6.1" % "provided"

and this is my scala file:

import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming._
import org.apache.spark.sql.types._
import org.apache.spark.sql.cassandra._

import com.datastax.oss.driver.api.core.uuid.Uuids
import com.datastax.spark.connector._


object StreamHandler {
    def main(args: Array[String]){

        val spark = SparkSession
            .builder
            .appName("Stream Handler")
            .config("spark.cassandra.connection.host","localhost")
            .getOrCreate()

        import spark.implicits._


        val Temp_DF = spark
            .read
            .cassandraFormat("train_temperature", "project")
            .load()
        
        Temp_DF.show(10)
    }
}

and this is the result:

Answer 1

Usually the problem is that when you do sbt package it builds a jar only with your code, and without dependencies. To mitigate this problem you have two approaches:

1. Specify Cassandra Connector when using spark-submit with --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 as described in documentation

2. Create a fat jar (with all necessary dependencies) using SBT assembly plugin , but you may need to take care of not including Spark classes into fat jar.