在python代码中使用hadoop fs -put命令将文件从本地文件系统传输到hdfs中的问题

Question

I am using this code in python which reads records from a file, does some processing on them and then write the outcome to a new file.Then I transfer the file from my local filesystem to hdfs:

read = open('file_read.txt', 'r')

for line in read:
    fields = line.split('|')
    columns.append(fields)

category = [-1,1,2,3,4,5,6]
out = open('file_write.txt', 'w')

for line in columns:
    out.write('{0}|{1}|{2}|{3}'.format(line[0], line[1], line[5], line[6].rstrip().replace('-','')))
    for val in category:
        if int(line[4]) == val:
            out.write('|{0}'.format(line[2]))
        else:
            out.write('|')
    for val in category:
        if int(line[4]) == val:
            out.write('|{0}'.format(line[3]))
        else:
            out.write('|')
    out.write('\n')
str = "HADOOP_USER_NAME=hdfs hadoop fs -put file_write.txt /folder1/folder2/"
result = system(str)

The problem is that during the transfer some of the last few records are getting lost from the file. The file that gets moved to hdfs has about 10 records less than the file which is on my local file system. I have tried -moveFromLocal also but same result occurs. Though if I execute the any of the above command from the terminal then complete file gets moved but its just when I am executing it from within a python script the issue comes.

Why is this issue coming and what could I do to resolve it?

UPDATE: The issue of missing records are coming only if I execute the part above the hadoop fs - put command. If I do not execute it and just move simple file then there is no loss of data occurring. I have tried to see if there any special character that is getting inserted and which may be causing the loss of last few records but couldn't find one (I tried to look for them by going through the file).

Answer 1

I can not reproduce the issue.

$ < /dev/urandom tr -dc "\n [:alnum:]" | head -c10000000 > test.txt
$ cat python_hdfs.py 
from os import system

str = "HADOOP_USER_NAME=hdfs hadoop fs -put test.txt /tmp/"
print system(str)
$ cat test.txt | wc -l
155682
$ python python_hdfs.py 
0
$ hadoop fs -cat /tmp/test.txt | wc -l
155682

Maybe config related?

• Is the exit status of the system call 0? Are you on linux or windows?
• How big is the file? Does it happen only with this specific one, or other files as well?