Java Apache Beam 测试管道用空值替换测试数据

Question

I have created a test pipeline like this:

        Pipeline pipeline;
        PipelineOptions pipelineOptions = TestPipeline.testingPipelineOptions();
        pipeline = Pipeline.create(pipelineOptions);

        FlattenLight flattenLight = new FlattenLight();
        DataflowMessage dataflowMessage = getTestDataflowMessage();
        
        PCollection<TableRow> flatttened = pipeline
                .apply("Create Input", Create.of(dataflowMessage))
                .apply(ParDo.of(flattenLight));

I want tot test the FlattenLight class, it is a DoFn child with a processElement(ProcessContext c) method.

The problem is that the test data generated with getTestDataflowMessage() does not goes through the pipeline. The FlattenLight object receives an Object with null values as fields.

The getTestDataflowMessage() creates fields as expected. You can see a lot of different test values are present:

debugger step at test data creation

But the FlattenLight class receives an Object that is mostly empty:

debugger step entering the FlattenLight object

As you can see there is not step between the data creation and the FlattenLight processing. Why does this happens? How to fix it?

Answer 1

I had the same issue. The solution was to add implements Serializable to all models within the model hierarchy. Take a closer look at your DataflowMessage, maybe you missed it somewhere.