Scala Apache Spark Cassandra表列表

Question

I want to list the tables of a keyspace in a Cassandra db using Apache Spark. I can access any cassandra table with sc.cassandraTable("keyspace", "table") but I'm not able to list all the tables in a keyspace, I want to loop over them. This is my code:

val conf = new SparkConf(true)
      .setAppName("Backup app").setMaster("local[4]")
      .set("spark.cassandra.connection.host", "XXXXX")
      .set("spark.cassandra.auth.username", "XXXX")
      .set("spark.cassandra.auth.password", "XXXXX")
      .setJars(Array("./lib/spark-cassandra-connector-assembly-2.0.2-39-g24f392d.jar"))

  val sc = new SparkContext(conf)

  sc.cassandraTable("keyspace", "userstable").select("salt").where("role = ?", "user").collect().toList.foreach {
    userkeyspace => println(userkeyspace)
  }

How can I do it?

Answer 1

I found the solution, here is the working code for me (I have a table stb.users where I store, among others, the keyspace of every user under the "salt" column):

val conf = new SparkConf(true)
      .setAppName("Backup app").setMaster("local[4]")
      .set("spark.cassandra.connection.host", "XXXX")
      .set("spark.cassandra.auth.username", "XXXX")
      .set("spark.cassandra.auth.password", "XXXX")
      .setJars(Array("./lib/spark-cassandra-connector-assembly-2.0.2-39-g24f392d.jar"))

  val sc = new SparkContext(conf)
  val sqlContext = new org.apache.spark.sql.SQLContext(sc);

  CassandraConnector(conf).withSessionDo { session =>
    sc.cassandraTable("stb", "users").select("salt").where("role = ?", "user").collect().toList.foreach {
      user =>
        val userSalt = user.getString("salt")
        val iterator = session.getCluster.getMetadata.getKeyspace(userSalt).getTables().iterator()
        while(iterator.hasNext) {
          val tableName = iterator.next().getName
          println(keyspace + " " + tableName)
        }
    }
  }

Answer 2

You can use

system.schema_columnfamilies

table, it contains list of tables in each keyspace.

SELECT keyspace_name, columnfamily_name FROM schema_columnfamilies;

or

sc.cassandraTable("system", "schema_columnfamilies").select("columnfamily_name").where("keyspace_name = ?", "the_keyspace")