從子進程調用的外部 python 腳本打印 tqdm 進度條

Question

我的主要目標是通過另一個 python 腳本（調用者腳本）中的subprocess進程運行外部 python 腳本（客戶端腳本）。 The console of the caller script displays all output from the client script except the tqdm output - so it is not a general problem of displaying output by subprocess , but a specific problem related to subprocess interacting with tqdm .

我的次要目標是我想了解它:)。 非常感謝周到的解釋。

客戶端腳本 (train.py) 包含幾個 tqdm 調用。 到目前為止，我還沒有看到各種 tqdm 參數配置之間的輸出有太大差異，所以讓我們使用最簡單的一個。

在train.py ：

...
from tqdm import tqdm

with tqdm(total = 10, ncols = 80,
          file=sys.stdout, position = 0, leave = True,
          desc='f5b: pbar.set_postfix') as pbar:
    for i in range(10):
        pbar.update(1)
        postfix = {'loss': '{0:.4f}'.format(1+i)}
        pbar.set_postfix(**postfix)
        sleep(0.1)

調用者腳本experiment.py執行function execute_experiment ，它通過參數command_list調用train.py ：

def execute_experiment(command_list):
    tic = time.time()
    try:
        process = subprocess.Popen(
            command_list, shell=False, 
            encoding='utf-8',
            bufsize=0,
            stdin=subprocess.DEVNULL,
            universal_newlines=True,
            stdout=subprocess.PIPE, 
            stderr=subprocess.PIPE
            )
        # Poll process for new output until finished
        # Source: https://stackoverflow.com/q/37401654/7769076
        while process.poll() is None:
            nextline = process.stdout.readline()
            sys.stdout.write(nextline)
            sys.stdout.flush()

    except CalledProcessError as err:
        print("CalledProcessError: {0}".format(err))
        sys.exit(1)

    except OSError as err:
        print("OS error: {0}".format(err))
        sys.exit(1)

    except:
        print("Unexpected error:", sys.exc_info()[0])
        raise

    if (process.returncode == 0):
        toc = time.time()
        time1 = str(round(toc - tic))
        return time1
    else:
        return 1

此腳本調用上面的 train.py 代碼片段確實返回 output 但 tqdm output 在 0 秒后停止，如下所示：

f5b: pbar.set_postfix:   0%|                             | 0/10 [00:00<?, ?it/s]
f5b: pbar.set_postfix:  10%|█▊                | 1/10 [00:00<00:00, 22310.13it/s]

腳本調用train.py原代碼返回除tqdm output外的所有output：

Training default configuration
train.py data --use-cuda ...
device: cuda
...

注釋：

1. shell = False ：由於 python 腳本調用 python 腳本。 當shell=True時，根本不調用客戶端腳本
2. bufsize=0 ：防止緩沖
3. train.py調用以sys.executable ，以確保在本地機器上調用相應 conda 環境的 python 解釋器。

問題：

1. tqdm.set_postfix是否阻止通過進度條 output 上游？ 我知道在調用tqdm.set_description時會發生這種情況，例如：

   pbar.set_description('已處理：%d' %(1 + i))

此代碼包含它：

def train(self, dataloader, max_batches=500, verbose=True, **kwargs):
    with tqdm(total=max_batches, disable=not verbose, **kwargs) as pbar:
        for results in self.train_iter(dataloader, max_batches=max_batches):
            pbar.update(1)
            postfix = {'loss': '{0:.4f}'.format(results['mean_outer_loss'])}

            if 'accuracies_after' in results:
                postfix['accuracy'] = '{0:.4f}'.format(
                    np.mean(results['accuracies_after']))
            pbar.set_postfix(**postfix)
    # for logging
    return results

2. 是嵌套的function調用進度條不顯示的原因嗎？

調用順序是train.py experiment.py nested.py 。

train.py通過以下方式在nested.py中調用火車function：

對於范圍內的紀元（args.num_epochs）：

results_metatraining = metalearner.train(meta_train_dataloader,
                  max_batches=args.num_batches,
                  verbose=args.verbose,
                  desc='Training',
                  # leave=False
                  leave=True
                  ) 

---

替代方案嘗試但沒有成功：

    ### try2
    process = subprocess.Popen(command_list, shell=False, encoding='utf-8',
                               stdin=DEVNULL, stdout=subprocess.PIPE)
    while True:
        output = process.stdout.readline().strip()
        print('output: ' + output)
        if output == '' and process.poll() is not None:  # end of output
            break
        if output: # print output in realtime
            print(output)
    else:
        output = process.communicate()
    process.wait()


    ### try6
    process = subprocess.Popen(command_list, shell=False,
                               stdout=subprocess.PIPE, universal_newlines=True)
    for stdout_line in iter(process.stdout.readline, ""):
        yield stdout_line 
    process.stdout.close()
    return_code = process.wait()
    print('return_code' + str(return_code))
    if return_code:
        raise subprocess.CalledProcessError(return_code, command_list)


    ### try7
    with subprocess.Popen(command_list, stdout=subprocess.PIPE, 
                          bufsize=1, universal_newlines=True) as p:
        while True:
            line = p.stdout.readline()
            if not line:
                break
            print(line)    
        exit_code = p.poll()

Answer 1

我認為 readline 正在等待'\n'，並且 tqdm 沒有創建新行，也許這會有所幫助（我沒有嘗試）：

import io
def execute_experiment(command_list):
    tic = time.time()
    try:
        process = subprocess.Popen(
            command_list, shell=False, 
            encoding='utf-8',
            bufsize=1,
            stdin=subprocess.DEVNULL,
            universal_newlines=True,
            stdout=subprocess.PIPE, 
            stderr=subprocess.STDOUT
            )
        # Poll process for new output until finished
        # Source: https://stackoverflow.com/q/37401654/7769076
        reader = io.TextIOWrapper(process.stdout, encoding='utf8')
        while process.poll() is None:
            char = reader.read(1)
            sys.stdout.write(char)
            sys.stdout.flush()

    except CalledProcessError as err:
        print("CalledProcessError: {0}".format(err))
        sys.exit(1)

    except OSError as err:
        print("OS error: {0}".format(err))
        sys.exit(1)

    except:
        print("Unexpected error:", sys.exc_info()[0])
        raise

    if (process.returncode == 0):
        toc = time.time()
        time1 = str(round(toc - tic))
        return time1
    else:
        return 1