如果 HDFS 中不存在目录，如何使用 pyspark 动态创建目录并设置文件和目录权限

Question

I am new to Hadoop can we create directory in hadoop dyanamically?

currently I am using below command:

hadoop fs -mkdir -p /data/test1/test2/test3/

and setting the file permission by using below command:

hdfs dfs -chmod -R 777 /path /data/test1/test2/test3/t_bill_sheet.csv

By Dyanamically I mean {year} and iteratively inside it folder by date like 5,6,7 etc.

Thanks in Advance

Answer 1

You can define a bash variable (or compute from current dates if you want) then reuse it over and over

YEAR=2000
MONTH=03
DAY=01
PATH="/data/$YEAR/$MONTH/$DAY"

hadoop fs -mkdir -p "$PATH"
hdfs dfs -chmod -R 777 /path "$PATH/t_bill_sheet.csv"

Answer 2

you can do it using a combination of exists() and mkdirs() methon in pyspark as below

fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration())

if not fs.exists(sc._jvm.org.apache.hadoop.fs.Path("path")): #returns true or false
    fs.mkdirs(spark._jvm.org.apache.hadoop.fs.Path("path"), FsPermission(777: Short) permission)