给定一个hdfs路径，我怎么知道它是一个文件夹还是一个带有python的文件

Question

I am not checking local file, I want to find out for a given string - whether it is a folder or a file on HDFS, in python.

For example, a string could be like:

hdfs://nameservice1/client/tdb_histscen_2/part-00001

It could be a file, or a folder that contains folder(s) and/or file(s)

Thank you very much.

Updated 20181105 as per suggestion from Jim Todd below:

hdfs://nameservice1/client/nova/scenarios/warehouse/pricetek_ibbk/tdb_histscen_asd/ doesn't exist at all

hdfs://nameservice1/client/nova/scenarios/warehouse/pricetek_ibbk/tdb_histscen_2 is a folder

As you can see below, the -test returns same result for them, what am I missing here?

Thank you.

[rxie@cedgedev03 code]$ hdfs dfs -test -e hdfs://nameservice1/client/nova/scenarios/warehouse/pricetek_ibbk/tdb_histscen_asd/
[rxie@cedgedev03 code]$ hdfs dfs -test -e hdfs://nameservice1/client/nova/scenarios/warehouse/pricetek_ibbk/tdb_histscen_2/
[rxie@cedgedev03 code]$ hdfs dfs -test -d hdfs://nameservice1/client/nova/scenarios/warehouse/pricetek_ibbk/tdb_histscen_2/
[rxie@cedgedev03 code]$ hdfs dfs -test -d hdfs://nameservice1/client/nova/scenarios/warehouse/pricetek_ibbk/tdb_histscen_asd/

Answer 1

There are several libraries to work with Hadoop in Python. For instance, if you use Pydoop , you can use pydoop.hdfs.path.isfile method.

You can check out their documentation

Answer 2

If you intention is to check if the URI is a directory or not using python, you can check alternatively like below:

import subprocess
location='hdfs://nameservice1/client/tdb_histscen_2/part-00001'

filexistchk="hdfs dfs -test -e "+location+";echo $?"
#echo $? will print the exit code of previously execited command
filexistchk_output=subprocess.Popen(filexistchk,shell=True,stdout=subprocess.PIPE).communicate()
filechk="hdfs dfs -test -d "+location+";echo $?"
filechk_output=subprocess.Popen(filechk,shell=True,stdout=subprocess.PIPE).communicate()
#Check if location exists
if '1' not in str(filexistchk_output[0]):
    #check if its a directory
    if '1' not in str(filechk_output[0]):
        print('The given URI is a directory: '+location)
    else:
        print('The given URI is a file: '+location)
else:
    print(location+ " does not exist. Please check the URI")

About the command: hdfs dfs -test -[ezd] URI

Options: The -e option will check to see if the file exists, returning 0 if true. The -z option will check to see if the file is zero length, returning 0 if true. The -d option will check to see if the path is directory, returning 0 if true. Example: hdfs dfs -test -d $yourdir