Hive在Python脚本中调用时尝试读取当前工作目录

Question

I am attempting to execute a Hive script from a Python wrapper. Part of code looks like

print(HiveArgs)
Hive = subprocess.Popen(HiveArgs, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
HiveOutput = Hive.communicate()

print("Out:" + HiveOutput[0])
print("=================================")
print("Err:" + HiveOutput[1])

The output of this is:

['hive', '-i ', '/edw/edwdev/tmp/spark.txn.init.tmp', '-f ', '/edw/edwdev/tmp/test.hql.tmp']
Out:
=================================
Err:
Logging initialized using configuration in file:/etc/hive/2.5.0.2-3/0/hive-log4j.properties
Exception in thread "main" java.io.FileNotFoundException: File file:/data/edw/edwdev/  does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:624)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:850)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:614)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:422)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:146)
        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:348)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:782)
        at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:427)
        at org.apache.hadoop.hive.cli.CliDriver.processInitFiles(CliDriver.java:439)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

where /data/edw/edwdev/ (file that Hive thinks is missing) is my working directory on a Linux server.

Changing working directory to script's location doesn't help. Using relative and absolute paths also makes no difference. If i copy values from the printed HiveArgs and execute the command from terminal ( hive -i /edw/edwdev/tmp/spark.txn.init.tmp -f /edw/edwdev/tmp/test.hql.tmp ), it works correctly.

What am I missing here?

Answer 1

Turned out that the issue was with Hive arguments. print(HiveArgs) line gave output:

['hive', '-i ', '/edw/edwdev/tmp/spark.txn.init.tmp', '-f ', '/edw/edwdev/tmp/test.hql.tmp']

The arguments passed are '-f ' and '-i ' (with trailing spaces) instead of '-f' and '-i' .

I am not sure what caused the issue within Hive leading it to read current working directory as some input file. Most likely Hive does not trim the arguments leading to this behavior. Removing the spaces fixed the issue.