[英]How to create groups of N elements from a PCollection Apache Beam Python
I am trying to accomplish something like this: Batch PCollection in Beam/Dataflow 我试图完成这样的事情: 在Beam / Dataflow中批量PCollection
The answer in the above link is in Java, whereas the language I'm working with is Python. 上面链接的答案是Java,而我正在使用的语言是Python。 Thus, I require some help getting a similar construction. 因此,我需要一些帮助来获得类似的结构。
Specifically I have this: 具体来说我有这个:
p = beam.Pipeline (options = pipeline_options)
lines = p | 'File reading' >> ReadFromText (known_args.input)
After this, I need to create another PCollection
but with a List
of N rows of "lines" since my use case requires a group of rows. 在此之后,我需要创建另一个PCollection
但有List
的“行”,因为我的使用情况下,N行需要一组行。 I can not operate line by line. 我无法逐行操作。
I tried a ParDo
Function using variables for count associating with the counter N rows and after groupBy
using Map
. 我尝试使用ParDo
函数,使用变量计数与计数器N行相关联,并在groupBy
之后使用Map
。 But these are reset every 1000 records, so it's not the solution I am looking for. 但是每1000条记录重置一次,所以这不是我要找的解决方案。 I read the example in the link but I do not know how to do something like that in Python. 我在链接中阅读了这个例子,但我不知道如何在Python中做这样的事情。
I tried saving the counters in Datastore, however, the speed difference between Dataflow reading and writing with Datastore is quite significant. 我尝试在Datastore中保存计数器,但是,Dataflow读取和写入数据存储之间的速度差异非常大。
What is the correct way to do this? 这样做的正确方法是什么? I don't know how else to approach it. 我不知道如何接近它。 Regards. 问候。
Assume the grouping order is not important, you can just group inside a DoFn
. 假设分组顺序不重要,您可以在DoFn
。
class Group(beam.DoFn):
def __init__(self, n):
self._n = n
self._buffer = []
def process(self, element):
self._buffer.append(element)
if len(self._buffer) == self._n:
yield list(self._buffer)
self._buffer = []
def finish_bundle(self):
if len(self._buffer) != 0:
yield list(self._buffer)
self._buffer = []
lines = p | 'File reading' >> ReadFromText(known_args.input)
| 'Group' >> beam.ParDo(Group(known_args.N)
...
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.