Using external classes with dataflow

Question

I am trying to make a custom transform class that takes in a side input and I am using PipelineTester and PAssert to try and test it, but I keep getting a no such method exception on methods I am trying to bring into the transform from other classes.

Caused by: java.lang.NoSuchMethodError: com.org.utils.MyUtils.createMap(Ljava/lang/Iterable;)Ljava/util/Map; at com.org.beam.MyTransform.ProcessElement(MyTransform.java:51)

I have tried using the @Autowired annotation to bring in the class like

@Autowired
private MyUtils myutils;

as well as just creating a static instance in the code like

private static MyUtils myUtils = new MyUtils();

and then calling

this.myUtils.createMap(mapThisToThat(inputCollection, this.myMap));

I have also tried making the methods static and calling them like

MyUtils.createMap(mapThisToThat(inputCollection, this.myMap));

the signature to mapThisToThat is

private Iterable<MyObject> mapThisToThat(Iterable<MyObject> objectIterator, Map<String, Integer> myMap) {

which is being passed into the createMap method which has this signature -

public Map<String, MyObject> createMap(Iterable<MyObject> inputCollection){

so it is passing in an Iterable of MyObjects correctly, but it says the method doesn't exist for some reason. does this mean beam transforms can't have external methods or am I doing something wrong?

Answer 1

For me in python, there are a variety of things I need to do for that to work:

https://cloud.google.com/dataflow/faq#how-do-i-handle-nameerrors https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/

For you in java, they don't have reciprocal documentation for some reason, but over here https://beam.apache.org/documentation/runners/dataflow/ they say things like this:

In some cases, such as starting a pipeline using a scheduler such as Apache AirFlow, you must have a self-contained application. You can pack a self-executing JAR by explicitly adding the following dependency on the Project section of your pom.xml, in addition to the adding existing dependency shown in the previous section.

In their examples readme https://github.com/mbrukman/apache-beam/tree/master/examples/java they say this

Alternatively, you may choose to bundle all dependencies into a single JAR and execute it outside of the Maven environment. For example, you can execute the following commands to create the bundled JAR of the examples and execute it both locally and in Cloud Platform

If you continue to browse that examples repo, there is a common folder with utils. Hopefully you can copy how they did it.