如何在Spark Streaming中使用Redis？

Question

我在java中使用火花流。我將sparkconfig obj配置為SparkConf sparkConf = new SparkConf().setAppName("MyApp").setMaster("local[2]") .set("spark.streaming.stopGracefullyOnShutdown","true") .set("redis.host", "localhost") .set("redis.port", "6379"); 並在JavastreamingContext中傳遞config obj。

JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.milliseconds(1000));

如何使用jssc對象訪問redis。 提前致謝。

Answer 1

這將從Redis列表中創建一個流

  SparkConf sparkConf = new SparkConf().setAppName("MyApp").setMaster("local[2]")
            .set("spark.streaming.stopGracefullyOnShutdown", "true")
            .set("redis.host", "localhost")
            .set("redis.port", "6379");


    JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.milliseconds(1000));

    RedisConfig redisConfig = new RedisConfig(new RedisEndpoint(sparkConf));

    RedisStreamingContext redisStreamingContext = new RedisStreamingContext(jssc.ssc());
    String[] keys = new String[]{"myList"};
    RedisInputDStream<Tuple2<String, String>> redisStream =
            redisStreamingContext.createRedisStream(keys, StorageLevel.MEMORY_ONLY(), redisConfig);

    redisStream.print();

    jssc.start();
    jssc.awaitTermination();

推送一些數據以列出：

LPUSH "myList" "aaaa"
LPUSH "myList" "bbbb"