在谷歌云中运行 Apache Beam 作业时找不到错误模块

Question

I am trying to run an Apache beam job in google cloud but is unsuccessful in completing it. I have tried debugging and other troubleshooting steps but it's still getting stuck every time, Here's the error:

  File "/home/avien/.pyenv/versions/dataflow/lib/python3.8/site-packages/apache_beam/transforms/core.py", line 1730, in <lambda>
    wrapper = lambda x: [fn(x)]
  File "xmlload.py", line 59, in <lambda>
NameError: name 'parse_into_dict' is not defined [while running 'parse-ptransform-73']

while running without the lamda function and directly passing it in beam.Map() it changed to:

File "/home/avien/.pyenv/versions/dataflow/lib/python3.8/site-packages/apache_beam/transforms/core.py", line 1730, in <lambda>
    wrapper = lambda x: [fn(x)]
  File "xmlload.py", line 36, in parse_into_dict
ModuleNotFoundError: No module named 'xmltodict' [while running 'parse-ptransform-73']

I have already setup pyenv and install xmltodict:

Requirement already satisfied: xmltodict in ./.pyenv/versions/3.8.13/envs/dataflow/lib/python3.8/site-packages (0.13.0)

Here the pipeline am trying to run:

import argparse
import logging
import apache_beam as beam
import xmltodict

def parse_into_dict(xmlfile):
    import xmltodict
    import apache_beam as beam
    with open(xmlfile) as ifp:
        doc = xmltodict.parse(ifp.read())
        return doc

table_schema = {
    'fields': [
        {'name' : 'CustomerID', 'type': 'STRING', 'mode': 'NULLABLE'},
        {'name' : 'EmployeeID', 'type': 'STRING', 'mode': 'NULLABLE'},
        {'name' : 'OrderDate', 'type': 'STRING', 'mode': 'NULLABLE'},
        {'name' : 'RequiredDate', 'type': 'STRING', 'mode': 'NULLABLE'},
        {'name' : 'ShipInfo', 'type': 'RECORD', 'mode': 'NULLABLE', 'fields': [
            {'name' : 'ShipVia', 'type': 'STRING', 'mode': 'NULLABLE'},
            {'name' : 'Freight', 'type': 'STRING', 'mode': 'NULLABLE'},
            {'name' : 'ShipName', 'type': 'STRING', 'mode': 'NULLABLE'},
            {'name' : 'ShipAddress', 'type': 'STRING', 'mode': 'NULLABLE'},
            {'name' : 'ShipCity', 'type': 'STRING', 'mode': 'NULLABLE'},
            {'name' : 'ShipRegion', 'type': 'STRING', 'mode': 'NULLABLE'},
            {'name' : 'ShipPostalCode', 'type': 'STRING', 'mode': 'NULLABLE'},
            {'name' : 'ShipCountry', 'type': 'STRING', 'mode': 'NULLABLE'},
            {'name' : 'ShippedDate', 'type': 'STRING', 'mode': 'NULLABLE'},
        ]},
    ]
}

def cleanup(x):
    import copy
    y = copy.deepcopy(x)
    if '@ShippedDate' in x['ShipInfo']: # optional attribute
        y['ShipInfo']['ShippedDate'] = x['ShipInfo']['@ShippedDate']
        del y['ShipInfo']['@ShippedDate']
    print(y)
    return y

def get_orders(doc):
    for order in doc['Root']['Orders']['Order']:
        yield cleanup(order)

def run(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument(
      '--output',
      required=True,
      help=(
          'Specify text file orders.txt or BigQuery table project:dataset.table '))

    known_args, pipeline_args = parser.parse_known_args(argv)
    with beam.Pipeline(argv=pipeline_args) as p:
        orders = (p
             | 'files' >> beam.Create(['orders.xml'])
             | 'parse' >> beam.Map(parse_into_dict)
             | 'orders' >> beam.FlatMap(get_orders))

        if '.txt' in known_args.output:
             orders | 'totxt' >> beam.io.WriteToText(known_args.output)
        else:
             orders | 'tobq' >> beam.io.WriteToBigQuery(known_args.output,
                                       schema=table_schema,
                                       write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND, #WRITE_TRUNCATE
                                       create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED)


if __name__ == '__main__':
    logging.getLogger().setLevel(logging.INFO)
    run()

I have tried the following steps till now:

1. tried to include all the functions inside pipeline itself and but the result is same.
2. Include all the imports in every function

Also, when running the parse_into_dict in a standalone python file it's not throwing any error at all am successfully able to convert xml to dict.

Any help is highly appreciated, Thanks in advance!

Answer 1

Try importing modules inside your function and pipeline definitions; or use --save_main_session . NameError s are common because the worker doesn't know objects defined in the global namespace.

Answer 2

In addition to @ningk answer, you have to give dataflow your orders.xml file. You are trying to load this file in the first step of your pipeline ( beam.Create['orders.xml'] ), however, dataflow does not know/have this file when it executes your pipeline.

Try adding a MANIFEST.in file (mind the caps) with the following content

include path/to/xml/orders.xml

in the source folder of your pipeline code. See here for an example file.

Answer 3

I had a similar problem with dependencies only with dataflow runner and it helped to include --requirements_file requirements.txt while running your script, so you end up with something like:

python pyscript.py --requirements_file requirements.txt

Check beam documentation https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/