apache beam 中的管道语法是如何实现的？

Question

I am now studying apache beam and out of curiosity, I would like to ask below question.

In advance, I have read below documents and threads.

https://beam.apache.org/documentation/programming-guide/#applying-transforms

Explain Apache Beam python syntax

I understand that pipe( | ) is the python version of .apply of java. However, I am curious to know how does python interpret __or__ operator as processor which processes each pcollection element that goes through from left to right.

I appreciate if someone could educate me and point me the reference of code.

Thanks, Yu

Answer 1

I would like to mark @Kolban 's reply as answer.

I did a Google search on "python operator overloading" and found a bunch of good references that seem likely. Searching the Github repository, it looks likely that this may be the actual code: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/ptransform.py#L470

Answer 2

It does so via operator overloading :

def __or__(self, right):
  """Used to compose PTransforms, e.g., ptransform1 | ptransform2."""
  if isinstance(right, PTransform):
    return _ChainedPTransform(self, right)
  return NotImplemented

Piping (|) is used to compose PTransforms , eg, ptransform1 | ptransform2 ptransform1 | ptransform2 .