How to start Spark without Cassandra in Datastax Enterprise

Question

I have recently installed DataStax Enterprise (v5.0) in my 5 node cluster. I am planning to use 3 nodes as spark and 2 nodes as cassandra multi-node cluster.

Now whenever I try to start spark in my spark nodes -

I get the following error -

sudo dse spark Exception in thread "main" java.io.IOException: Failed to open native connection to Cassandra at {127.0.0.1}:9042 at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:160)

Why does spark look for cassandra ? Is there no way to start spark without cassandra in DSE ?

Also which conf files in DSE should i change so that spark connects to cassandra running on another node (and not the one running locally at 127.0.0.1:9042)

Answer 1

You cannot currently (5.0) start a DSE node without Cassandra. But just because you have to start Cassandra doesn't mean you have to use it. Just make sure that you are always using Network Topology Strategy and DSE will only use C* on those machines for gossip and schema information.

Also for a note dse spark just starts the spark shell. It doesn't start the Spark Master and Worker infrastructure.