在 scala 中使用 spark 流从文件夹流式传输时，如何读取包括子文件夹在内的所有文件？

Question

I have some files that I want to stream using spark structured streaming. The structure is something like this:

myFolder
└── subFolderOne
    ├── fileOne.gz
    ├── fileTwo.gz
    └── fileThree.gz
└── subFolderTwo
    ├── fileFour.gz
    ├── fileFive.gz
    ├── fileSix.gz

When i only do the following, it works:

val df = spark
  .readStream
  .format("json")
  .schema(schema)
  .option("maxFilesPerTrigger", 1)
  .json("/myFolder/subFolderOne/")     <-------

but I want to read it at the root level: /myFolder/ so that it picks all files within any number of sub folders. Is this possible?

I am using spark 2.4.5 and scala 2.11.6

Answer 1

So, it turns out to be as simple as this:

Before:

.json("/myFolder/") 

After

.json("/myFolder/*")