[英]Dataflow - XML Source - Python - How?
I'm trying to source an XML file into my dataflow code.我正在尝试将 XML 文件导入到我的数据流代码中。 I see java has a built in XMLIo but Python doesn't?我看到 java 有一个内置的 XMLIo 但 Python 没有? I'm also struggling to understand what are the initial steps to ParDo/DoFn it myself.我也很难理解自己 ParDo/DoFn 的初始步骤是什么。 This is an example of the XML file below.这是以下 XML 文件的示例。 My pipeline below when parsing .csv I understand but I'm not understanding how to start with an XML source.下面解析 .csv 时的管道我理解,但我不理解如何从 XML 源开始。 Do I need to manually create a PCollection and go from there?我是否需要手动创建一个 PCollection 并从那里开始?
My goal is to return each element as a tuple.我的目标是将每个元素作为一个元组返回。 The key would be the country name and each element after (in a nested array) would be the values.键是国家名称,之后(在嵌套数组中)的每个元素都是值。
<?xml version="1.0"?>
<data>
<country name="Liechtenstein">
<rank>1</rank>
<year>2008</year>
<gdppc>141100</gdppc>
<neighbor name="Austria" direction="E"/>
<neighbor name="Switzerland" direction="W"/>
</country>
<country name="Singapore">
<rank>4</rank>
<year>2011</year>
<gdppc>59900</gdppc>
<neighbor name="Malaysia" direction="N"/>
</country>
<country name="Panama">
<rank>68</rank>
<year>2011</year>
<gdppc>13600</gdppc>
<neighbor name="Costa Rica" direction="W"/>
<neighbor name="Colombia" direction="E"/>
</country>
</data>
def run():
argv = [
'--project={0}'.format(PROJECT),
'--staging_location=gs://{0}/'.format(BUCKET),
'--temp_location=gs://{0}/'.format(BUCKET),
'--runner=DataflowRunner'
#'--runner=DirectRunner'
]
p = beam.Pipeline(argv=argv)
(p
| 'ReadFromGCS' >> beam.io.textio.ReadFromText('gs://{0}/example.csv'.format(BUCKET))
-[SNIP]-
The code below will collect each country information.下面的代码将收集每个国家/地区的信息。
The output is a list of tuples.输出是一个元组列表。
The first element in the tuple is the country name and the second element is a list of the other country properties.元组中的第一个元素是国家名称,第二个元素是其他国家属性的列表。
import xml.etree.ElementTree as ET
xml = '''<?xml version="1.0"?>
<data>
<country name="Liechtenstein">
<rank>1</rank>
<year>2008</year>
<gdppc>141100</gdppc>
<neighbor name="Austria" direction="E"/>
<neighbor name="Switzerland" direction="W"/>
</country>
<country name="Singapore">
<rank>4</rank>
<year>2011</year>
<gdppc>59900</gdppc>
<neighbor name="Malaysia" direction="N"/>
</country>
<country name="Panama">
<rank>68</rank>
<year>2011</year>
<gdppc>13600</gdppc>
<neighbor name="Costa Rica" direction="W"/>
<neighbor name="Colombia" direction="E"/>
</country>
</data>'''
result = []
root = ET.fromstring(xml)
for country in root.findall('.//country'):
result.append((country.attrib['name'],[x.text if x.text else x.attrib for x in list(country)]))
print(result)
output输出
[('Liechtenstein', ['1', '2008', '141100', {'name': 'Austria', 'direction': 'E'}, {'name': 'Switzerland','direction': 'W'}]), ('Singapore', ['4', '2011', '59900', {'name': 'Malaysia', 'direction': 'N'}]), ('Panama', ['68', '2011', '13600', {'name': 'Costa Rica', 'direction': 'W'}, {'name': 'Colombia', 'direction': 'E'}])]
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.