[英]Running python scripts with variable parser arguments from Jupyter Notebook
I'm Working on transformers .我正在研究变压器。
I have a python script there, that takes arguments as input via argparse.我在那里有一个 python 脚本,它通过 argparse 将参数作为输入。
Here is part of it:这是其中的一部分:
parser = argparse.ArgumentParser()
parser.add_argument("--model_type", default=None, type=str, required=True,
help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()))
parser.add_argument("--model_name_or_path", default=None, type=str, required=True,
help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS))
I want to be able to call the script iteratively with different arguments.我希望能够使用不同的参数迭代调用脚本。 I can call the script with %run or !python < bash command >, but can I cannot pass variables to it be interpreted as arguments, because it treats the variables as the actual string value:我可以使用 %run 或 !python < bash command > 调用脚本,但是我不能将变量传递给它以解释为参数,因为它将变量视为实际的字符串值:
%run examples/run_lm_finetuning.py --gradient_accumulation_steps=1 --output_dir='output_medium' --model_type='gpt2' \
--model_name_or_path=model_load --do_train --train_data_file='/root/sharedfolder/omri/data/pieces/backup/{}'.format(file)\
--overwrite_output_dir --per_gpu_train_batch_size=1 --per_gpu_eval_batch_size=1 --save_total_limit=5
Returns:返回:
OSError: Model name 'model_load' was not found in
model name list (gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2).
We assumed 'model_load' was a path or url to a configuration file named
config.json or a directory containing such a file but couldn't find any
such file at this path or url.
It looks like {}
expands variables.看起来{}
扩展了变量。 I stumbled on it trying do the Python f''
formatting.我偶然发现它尝试执行 Python f''
格式。 I don't see it in %run
docs;我在%run
docs 中没有看到它; it must be part of the %magic syntax.它必须是 %magic 语法的一部分。
Anyways, with a simple echo
script:无论如何,使用一个简单的echo
脚本:
In [29]: cat echo.py
import sys
print(sys.argv)
In [30]: foo = "a string"
In [31]: run echo.py {foo} bar
['echo.py', 'a', 'string', 'bar']
In [32]: run echo.py "{foo}" bar
['echo.py', 'a string', 'bar']
=== ===
With another magic用另一种魔法
In [71]: astr="*.h5"
In [72]: ls {astr}
abc_copy.h5 string.h5 testdate.h5 test_str.h5...
=== ===
$
also does this: $
也这样做:
In [79]: foo = "a string"
In [80]: run echo.py $foo bar
['echo.py', 'a', 'string', 'bar']
How to pass a variable to magic ´run´ function in IPython 如何将变量传递给 IPython 中的魔法“运行”函数
IPython.core.magic.no_var_expand(magic_func) Mark a magic function as not needing variable expansion IPython.core.magic.no_var_expand(magic_func) 将魔法函数标记为不需要变量扩展
By default, IPython interprets {a} or $a in the line passed to magics as variables that should be interpolated from the interactive namespace before passing the line to the magic function.默认情况下,IPython 将传递给 magics 的行中的 {a} 或 $a 解释为变量,这些变量应该在将行传递给 magic 函数之前从交互式命名空间内插。 This is not always desirable, eg when the magic executes Python code (%timeit, %time, etc.).这并不总是可取的,例如当魔法执行 Python 代码时(%timeit、%time 等)。 Decorate magics with @no_var_expand to opt-out of variable expansion.使用 @no_var_expand 装饰魔法以选择退出变量扩展。
https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.magic.html https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.magic.html
RafalS's answer is great, but doesn't print stdout if you want, here is how: RafalS 的回答很棒,但如果您愿意,不会打印标准输出,方法如下:
Put the entire statement in an fstring and assign to a variable run :将整个语句放在 fstring 中并分配给变量run :
run = f"python examples/run_lm_finetuning.py\
--model_name_or_path={model_load}\
--train_data_file='/root/sharedfolder/omri/data/pieces/backup/{file}'\
--overwrite_output_dir\
--per_gpu_train_batch_size=1\
--per_gpu_eval_batch_size=1\
--save_total_limit=5"
Then run through Jupyter shell by simply:然后通过简单的方式运行 Jupyter shell:
!{run}
So model_load
in %run
should be interpreted as python variable?那么%run
model_load
应该被解释为 python 变量吗? That would be a little weird, don't you think?那会有点奇怪,你不觉得吗? Try calling python directly from python, not through ipython magic functions:尝试直接从 python 调用 python,而不是通过 ipython 魔术函数:
In [18]: import subprocess
In [19]: model_load = "gpt2"
In [20]: subprocess.run(f"python file.py --model_name_or_path={model_load}", shell=True)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.