如何从 S3 的 Pyspark 子文件夹中创建一个新的 dataframe 和 CSV 文件

Question

Hi I'm very new to Pyspark and S3. I have problem at hand. I have a folder, which consists of subfolders and files and also files from the subfolder(all CSVs) i need to create a new dataframe or a csv file where i get contents of the files and create as a single file. Which later need to be read to a table in postgress

Can anyone please help me. I have code in python, but not sure how to go about with pyspark and S3

Answer 1

Try with this option .

recursiveFileLookup – recursively scan a directory for files. Using this option disables partition discovery.

df = spark.read.option("header","true").option("recursiveFileLookup","true").csv("s3://path/to/root/")