从AWS s3中的文件夹发送Spark流-PySpark

Question

I have a folder "input" in aws 3. I want to stream the new files created in that folder. I am using this code:

ssc = StreamingContext(sc, 10)
rdd = ssc.textFileStream("s3n://dynamodbstream607/input/*")

But it throws an error saying that input/* doesnot exist. How do i get spark to stream from a folder in S3

Answer 1

根据此PR ，它将在2.0中可用。