[英]Spark Structured Streaming Kinesis Data source
Is it possible to use Kinesis streams as a data source for Spark structured streaming?是否可以将 Kinesis 流用作 Spark 结构化流的数据源? I can't find any connector available.
我找不到任何可用的连接器。
Qubole have a kinesis-sql library for exactly this. Qubole 有一个 kinesis-sql 库就是为了这个。
https://github.com/qubole/kinesis-sql https://github.com/qubole/kinesis-sql
Then you can use the source similar to any other Spark Structured Streaming source:然后您可以使用类似于任何其他 Spark Structured Streaming 源的源:
val source = spark
.readStream
.format("kinesis")
.option("streamName", "spark-source-stream")
.option("endpointUrl", "https://kinesis.us-east-1.amazonaws.com")
.option("awsAccessKeyId", [YOUR_AWS_ACCESS_KEY_ID])
.option("awsSecretKey", [YOUR_AWS_SECRET_KEY])
.option("startingPosition", "TRIM_HORIZON")
.load
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.