How does Celery actually execute the task?

Question

I have a text file inside directory the_files that contains several line like this

aaaaabbbb cccc--ddddeee ffff
gggjjjkkk eers--kklliii kkll
...

I wrote a celery script that manipulate each line in the text file.

from celery import Celery

import os

app = Celery('tasks', broker='amqp://guest@localhost//')

path = "the_files/"

@app.task
def do_task_txt():
    dir_path = os.listdir(path)
    for file in dir_path:
        if file.endswith(".txt"):
            f = open(path + file, "r")
            for line in f:
                string1 = line[0:14].replace(" ", "")
                string2 = line[16:].replace(" ", "")

            #print string1, string2
            return string1, string2
        f.close()

When I run this script with celery it provides the following result

[tasks]
  . tasks.do_task_txt

[2017-03-22 13:51:00,713: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2017-03-22 13:51:00,791: INFO/MainProcess] mingle: searching for neighbors
[2017-03-22 13:51:01,966: INFO/MainProcess] mingle: all alone
[2017-03-22 13:51:02,055: INFO/MainProcess] celery@Ling-Air ready.
[2017-03-22 13:51:25,624: INFO/MainProcess] Received task: tasks.do_task_txt[77166520-21a8-466e-9522-cb2b1821a185]  
[2017-03-22 13:51:26,152: INFO/PoolWorker-2] Task tasks.do_task_txt[77166520-21a8-466e-9522-cb2b1821a185] succeeded in 0.00866508999752s: ('aaaaabbbbcccc', 'ddddeeeffff')

It showed the first line only.

I was hoping to get it to show for every line perhaps like this maybe?

[2017-03-22 13:51:26,152: INFO/PoolWorker-2] Task tasks.do_task_txt[77166520-21a8-466e-9522-cb2b1821a185] succeeded in 0.00866508999752s: ('aaaaabbbbcccc', 'ddddeeeffff'),('gggjjjkkkeers', 'kklliiikkll'),(.....,...)

I checked my script by calling print string1, string2 and it did print the result as what I expected like this

aaaaabbbbcccc ddddeeeffff
gggjjjkkkeers kklliiikkll
...

My question is how does Celery execute the task? When I execute the task do_task_txt , it shows only one line from the file that has been manipulated. How do I show all of the line that has been manipulated instead of one line only?

Thank you for your suggestion.

Answer 1

first f.close() is never called when inside the if-statement, so file handle will remain.

Second your task returns a tuple which celery (or the broker) do not parse correctly.

Return a merged string instead:

return ", ".join(string1, string2)

Answer 2

When you return , the function execution is finished, it won't process left lines and other files. You should save results and return finally.

@app.task
def do_task_txt():
    dir_path = os.listdir(path)
    result = []
    for file in dir_path:
        if file.endswith(".txt"):
            f = open(path + file, "r")
            for line in f:
                string1 = line[0:14].replace(" ", "")
                string2 = line[16:].replace(" ", "")

            #print string1, string2
            result += [string1, string2]
        f.close()

    return result