用户定义的聚合函数 Spark Java - 合并问题

Question

I'm trying to use a User Defined Aggregate Function following the documentation here , I want to first pass 2 values, x and y, to a SimpleRegression then merge the simpleRegression by doing append. My problem is that the reduce function receives the values correctly (if I ask it to print x and y it prints them correctly), however if I see how many values have been added to the regressors in the merge function (using the getN() method as shown in the code provided) it returns me that no value has been added to those regressors, that is, as if no addData() was performed on them, why does this happen? What am I doing wrong?

Clearly this does not allow me to do what I want: to obtain the slope and the intercept of each regression line, because when the finish function is executed the regressors are empty and therefore slope and intercept are set to NaN.

This is my code (Java):

public static class RegressorAggregator extends Aggregator<Tuple2<Long, Long>, SimpleRegressionWrapper, LineParameters> {


    //Valore zero per l'aggregazione - dovrebbe soddisfare a+zero=a;
    public SimpleRegressionWrapper zero(){
        return new SimpleRegressionWrapper();
    }

    public SimpleRegressionWrapper reduce(SimpleRegressionWrapper simpleRegression, Tuple2<Long, Long> xy){
        double x = (double)xy._1;
        double y = (double)xy._2;
        simpleRegression.addData(x,y);
        return simpleRegression;
    }

    public SimpleRegressionWrapper merge(SimpleRegressionWrapper a, SimpleRegressionWrapper b){
        Logger log = LogManager.getLogger(getClass().getSimpleName());
        log.error(a.getN() + " " + b.getN());
        a.append(b);
        return a;
    }

    public LineParameters finish(SimpleRegressionWrapper simpleRegression){
        return new LineParameters(simpleRegression.getSlope(), simpleRegression.getIntercept());
    }

    public Encoder<SimpleRegressionWrapper> bufferEncoder(){
        return Encoders.bean(SimpleRegressionWrapper.class);
    }

    public Encoder<LineParameters> outputEncoder(){
        return Encoders.bean(LineParameters.class);
    }

}

Answer 1

The problem can be resolved changing this:

public Encoder<SimpleRegressionWrapper> bufferEncoder(){
        return Encoders.bean(SimpleRegressionWrapper.class);
    }

into this:

public Encoder<SimpleRegressionWrapper> bufferEncoder(){
        return Encoders.javaSerialization(SimpleRegressionWrapper.class);
    }