Pyspark：从路径读取多个 JSON 文件

Question

I am trying to read 20 JSON files from a path and create a dataframe, but even though the schema gets created, the df only contains null values ("Query returned no results"). Here is my code:

ordersJsonPath = "dbfs:/user/xyz/dbacademy/raw/orders/stream" 
ordersDF = spark.read.schema(userDefinedSchema).json(ordersJsonPath)

When I run the code with a specific JSON file, it works.

"dbfs:/user/xyz/dbacademy/raw/orders/stream/order_0612a18b-0cc7-43ea-9f5b-155aad967cb9_2020-01-01.json"

From what I understood, I need to create a schema manually when working with JSON or am I confusing things here and a manual schema is obsolete when working with multiple files?

Thank you in advance!

Edit: I checked the schema of the first 5 files in the path, they are all the same and they can all be read by the query I wrote (not streamed though, as this requires to give the whole path as input). When I start the readStream query, not even these first 5 JSON files get processed. Just nothing happens:

Another edit: I got it solved, needed to add /* to the path ...

Answer 1

解决了，需要将/*添加到路径...