Spark Streaming 使用 JDBC 源和 Redis Stream

Question

I'm trying to build a little mix of technologies to implement a solution on my work. Since I'm new to most of them, sometimes I got stuck, but could find solution to some of the problems I'm facing. Right now, both objects are running on Spark, but I can't seem to identify why the Streaming are not working.

Maybe is the way redis implements its sink on the writing to stream side, maybe is the way I'm trying to do the job. Almost all of the examples I found on streaming are related to Spark samples, like streaming text or TCP, and the only solution I found on relational databases are based on kafka connect, which I can't use right now because the company doesn't have the Oracle option to CDC on Kafka.

My scenario is as follows. Build a Oracle -> Redis Stream -> MongoDB Spark application.

I've built my code based on the examples of spark redis And used the sample code to try implement a solution to my case. I load the Oracle data day by day and send to a redis stream which later will be extracted from the stream and saved to Mongo. Right now the sample below is just trying to remove from the stream and show on console, but nothing is shown.

The little 'trick' I've tried was to create a CSV directory, read from it, and later grab the date from the csv and use to query the oracle db, then saving the oracle DataFrame on redis with the foreachBatch command. The data is saved, but I think not in the right way, because using the sample code to read the stream nothing is received.

Those are the codes:

** Writing to Stream **

object SendData extends App {
  Logger.getLogger("org").setLevel(Level.INFO)
  val oracleHost = scala.util.Properties.envOrElse("ORACLE_HOST", "<HOST_IP>")
  val oracleService = scala.util.Properties.envOrElse("ORACLE_SERVICE", "<SERVICE>")
  val oracleUser = scala.util.Properties.envOrElse("ORACLE_USER", "<USER>")
  val oraclePwd = scala.util.Properties.envOrElse("ORACLE_PWD", "<PASSWD>")
  val redisHost = scala.util.Properties.envOrElse("REDIS_HOST", "<REDIS_IP>")
  val redisPort = scala.util.Properties.envOrElse("REDIS_PORT", "6379")
  val oracleUrl = "jdbc:oracle:thin:@//" + oracleHost + "/" + oracleService
  val userSchema = new StructType().add("DTPROCESS", "string")
  val spark = SparkSession
    .builder()
    .appName("Send Data")
    .master("local[*]")
    .config("spark.redis.host", redisHost)
    .config("spark.redis.port", redisPort)
    .getOrCreate()
  val sc = spark.sparkContext
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  import sqlContext.implicits._
  val csvDF = spark.readStream.option("header", "true").schema(userSchema).csv("/tmp/checkpoint/*.csv")
  val output = csvDF
    .writeStream
    .outputMode("update")
    .foreachBatch {(df :DataFrame, batchId: Long) => {
      val dtProcess = df.select(col("DTPROCESS")).first.getString(0).take(10)
      val query = s"""
        (SELECT 
            <FIELDS>
        FROM 
            TABLE
        WHERE
          DTPROCESS BETWEEN (TO_TIMESTAMP('$dtProcess 00:00:00.00', 'YYYY-MM-DD HH24:MI:SS.FF') + 1)
          AND (TO_TIMESTAMP('$dtProcess 23:59:59.99', 'YYYY-MM-DD HH24:MI:SS.FF') + 1)
        ) Table
      """
      val df = spark.read
        .format("jdbc")
        .option("url", oracleUrl)
        .option("dbtable", query)
        .option("user", oracleUser)
        .option("password", oraclePwd)
        .option("driver", "oracle.jdbc.driver.OracleDriver")
        .load()
      df.cache()
      if (df.count() > 0) {
        df.write.format("org.apache.spark.sql.redis")
          .option("table", "process")
          .option("key.column", "PRIMARY_KEY")
          .mode(SaveMode.Append)
          .save()
      }
      if ((new DateTime(dtProcess).toLocalDate()).equals(new LocalDate()))
        Seq(dtProcess).toDF("DTPROCESS")
          .coalesce(1)
          .write.format("com.databricks.spark.csv")
          .mode("overwrite")
          .option("header", "true")
          .save("/tmp/checkpoint")
      else {
        val nextDay = new DateTime(dtProcess).plusDays(1)
        Seq(nextDay.toString(DateTimeFormat.forPattern("YYYY-MM-dd"))).toDF("DTPROCESS")
          .coalesce(1)
          .write.format("com.databricks.spark.csv")
          .mode("overwrite")
          .option("header", "true")
          .save("/tmp/checkpoint")
        }
      }}
    .start()
  output.awaitTermination()
}

** Reading from Stream **

object ReceiveData extends App {
  Logger.getLogger("org").setLevel(Level.INFO)
  val mongoPwd = scala.util.Properties.envOrElse("MONGO_PWD", "bpedes")
  val redisHost = scala.util.Properties.envOrElse("REDIS_HOST", "<REDIS_IP>")
  val redisPort = scala.util.Properties.envOrElse("REDIS_PORT", "6379")
  val spark = SparkSession
    .builder()
    .appName("Receive Data")
    .master("local[*]")
    .config("spark.redis.host", redisHost)
    .config("spark.redis.port", redisPort)
    .getOrCreate()
  val processes = spark 
    .readStream
    .format("redis")
    .option("stream.keys", "process")
    .schema(StructType(Array(
      StructField("FIELD_1", StringType),
        StructField("PRIMARY_KEY", StringType),
      StructField("FIELD_3", TimestampType),
      StructField("FIELD_4", LongType),
      StructField("FIELD_5", StringType),
      StructField("FIELD_6", StringType),
      StructField("FIELD_7", StringType),
      StructField("FIELD_8", TimestampType)
    )))
    .load()
  val query = processes
    .writeStream
    .format("console")
    .start()
  query.awaitTermination()
}

Answer 1

This code writes the dataframe to Redis as hashes (not to the Redis stream).

df.write.format("org.apache.spark.sql.redis")
          .option("table", "process")
          .option("key.column", "PRIMARY_KEY")
          .mode(SaveMode.Append)
          .save()

Spark-redis doesn't support writing to Redis stream out of the box.