[英]Read/Mount a csv file inside train.py of Azure ML Pipeline
我們正在從 Eventhub 和 AppInsight 收集數據並將其存儲在 azure blob 中。 通過使用 AzureML 管道,我想通過兩種不同的邏輯(一個用於 ml,另一個用於欺詐分析)將我的數據集傳遞到 train.py。
但我無法從train.py
內部讀取 csv 文件以進行進一步處理
這是我的train.py
,它通過 Azure 機器學習管道中的PythonScriptStep
運行
import argparse
import os
import pandas as pd
print("In train.py")
parser = argparse.ArgumentParser("train")
parser.add_argument("--input_data", type=str, help="input data")
parser.add_argument("--output_train", type=str, help="output_train directory")
args = parser.parse_args()
print("Argument 1: %s" % args.input_data)
df = pd.read_csv(args.input_data)
print(df.head())
print("Argument 2: %s" % args.output_train)
if not (args.output_train is None):
os.makedirs(args.output_train, exist_ok=True)
print("%s created" % args.output_train)
這是運行流水線的代碼
ws = Workspace.from_config()
def_blob_store = Datastore(ws, "basic_data_store")
aml_compute_target = "test-cluster"
try:
aml_compute = AmlCompute(ws, aml_compute_target)
print("found existing compute target.")
except ComputeTargetException:
print("Error")
source_directory = './train'
blob_input_data = DataReference(
datastore=def_blob_store,
data_reference_name="device_data",
path_on_datastore="_fraud_data/test.csv")
trainStep = PythonScriptStep(
script_name="train.py",
arguments=["--input_data", blob_input_data, "--output_train", processed_data1],
inputs=[blob_input_data],
outputs=[processed_data1],
compute_target=aml_compute,
source_directory=source_directory,
runconfig=run_config
)
pipeline1 = Pipeline(workspace=ws, steps=[compareStep])
pipeline_run1 = Experiment(ws, 'Data_dependency').submit(pipeline1)
在輸出跟蹤的下方,您可以看到輸出Argument 1
正在打印文件的路徑
Argument 1: /mnt/batch/tasks/shared/LS_root/jobs/pipeline-shohoz/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/mounts/basic_data_store/_fraud_data/test.csv
所以我已經成功地通過了數據集,但無法在pd.read_csv(args.input_data)
線上讀取 train.py 中的文件。 它正在顯示
FileNotFoundError: [Errno 2] File b'/mnt/batch/tasks/shared/LS_root/jobs/pipeline-shohoz/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/mounts/basic_data_store/_fraud_data/test.csv'
這是從完整的跟蹤70_driver_log.txt
我從azureml日志下載,
Preparing to call script [ train.py ] with arguments: ['--input_data', '/mnt/batch/tasks/shared/LS_root/jobs/pipeline-shohoz/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/mounts/basic_data_store/_fraud_data/test.csv', '--output_train', '/mnt/batch/tasks/shared/LS_root/jobs/pipeline-shohoz/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/mounts/basic_data_store/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/processed_data1']
After variable expansion, calling script [ train.py ] with arguments: ['--input_data', '/mnt/batch/tasks/shared/LS_root/jobs/pipeline-shohoz/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/mounts/basic_data_store/_fraud_data/test.csv', '--output_train', '/mnt/batch/tasks/shared/LS_root/jobs/pipeline-shohoz/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/mounts/basic_data_store/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/processed_data1']
In train.py
Argument 1: /mnt/batch/tasks/shared/LS_root/jobs/pipeline-shohoz/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/mounts/basic_data_store/_fraud_data/test.csv
The experiment failed. Finalizing run...
Cleaning up all outstanding Run operations, waiting 300.0 seconds
1 items cleaning up...
Cleanup took 0.001172780990600586 seconds
Starting the daemon thread to refresh tokens in background for process with pid = 136
Traceback (most recent call last):
File "train.py", line 18, in <module>
df = pd.read_csv(args.input_data) #str()
File "/azureml-envs/azureml_eb042e80b9a6abdb5821a78683153a38/lib/python3.6/site-packages/pandas/io/parsers.py", line 685, in parser_f
return _read(filepath_or_buffer, kwds)
File "/azureml-envs/azureml_eb042e80b9a6abdb5821a78683153a38/lib/python3.6/site-packages/pandas/io/parsers.py", line 457, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/azureml-envs/azureml_eb042e80b9a6abdb5821a78683153a38/lib/python3.6/site-packages/pandas/io/parsers.py", line 895, in __init__
self._make_engine(self.engine)
File "/azureml-envs/azureml_eb042e80b9a6abdb5821a78683153a38/lib/python3.6/site-packages/pandas/io/parsers.py", line 1135, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/azureml-envs/azureml_eb042e80b9a6abdb5821a78683153a38/lib/python3.6/site-packages/pandas/io/parsers.py", line 1917, in __init__
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 382, in pandas._libs.parsers.TextReader.__cinit__
File "pandas/_libs/parsers.pyx", line 689, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File b'/mnt/batch/tasks/shared/LS_root/jobs/pipeline-shohoz/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/mounts/basic_data_store/_fraud_data/test.csv' does not exist: b'/mnt/batch/tasks/shared/LS_root/jobs/pipeline-shohoz/azureml/d92be2ab-e63f-4883-a14b-a64fa5bb431d/mounts/basic_data_store/_fraud_data/test.csv'
我試過相對路徑
azureml/8d2b7bee-6cc5-4c8c-a685-1300a240de8f/mounts/basic_data_store/_fraud_data/test.csv
還有烏里
wasbs://shohoz-container@shohozds.blob.core.windows.net/azureml/azureml/8d2b7bee-6cc5-4c8c-a685-1300a240de8f/mounts/basic_data_store/_fraud_data/test.csv
但以相同的FileNotFoundError
結果結束。 在過去的 3-4 天里,我一直在用頭撞牆。 任何幫助都會拯救我的大腦。
您可以使用 PipelineDataset 對象在 PythonScriptStep 中包含已注冊的數據集 - 請參閱https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedataset?view= azure-ml-py了解更多細節和示例。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.