Register Anonymous class functionality

Question

While writing spark code, I'm using UDF (user defined function). UDF is an interface and its impelemented in in below way.

package sparkProject;

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.api.java.UDF1;
import org.apache.spark.sql.types.DataTypes;

public class UDFfunctions {

    public static void registerCountryCodeFunction(SparkSession spark) {

        spark.udf().register("registerCountryCodeFunctionUDF", new UDF1<String, Integer>() {

            /**
             * 
             */
            private static final long serialVersionUID = 1L;

            @Override
            public Integer call(String t1) throws Exception {
                if (t1.toString().toUpperCase().startsWith("I")) {

                    return 01;
                }
                return 02;
            }

        }, DataTypes.IntegerType);

    }

}

So UDF1 is an interface and having call method in it. When I create an instance using UDF1 interface, JVM creates Anonymus class which implements UDF1 interface and implements call method too.

In SPARK Api, to use registerCountryCodeFunctionUDF functionality, I need to perform below operation

UDFfunctions.registerCountryCodeFunction(spark);
ds2_populationGt100k_with_ia_filters_only.withColumn("countryCode", callUDF("registerCountryCodeFunctionUDF",ds2_populationGt100k_with_ia_filters_only.col("countryName")));

I want to know, How can I create my own UDF function, where user could register it and then use it in Java ...?

In Simple words, How can I create kind of code in Java where when we call callUDF with required parameters, it calls the functionality which is written Anonymus class..? What needs to written so that when callUDF gets called, it calls the same functionality which is written Anonymus class ?

I hope my question is clear to all..!

Thanks

Answer 1

Whether an object is an instance of anonymous class or not doesn't change anything to how you use it and call its methods.

Your framework simply stores the instances of UDF in a Map somewhere, indexed by the name you provide. And the callUDF() method simply gets it from the Map and invokes its call() method.

Here is a complete example doing the same thing:

// similar to UDF, but not generic to make it easier to understand
interface Callback {
    void call(String message);
}

class Registry {
    private Map<String, Callback> callbacks = new HashMap<>();

    public registerCallback(String name, Callback callback) {
        callbacks.put(name, callback;
    }

    public void invokeCallback(String name, String message) {
        Callback cb = map.get(name);
        cb.call(message);
    }
}

And now you can create Callback instances, using a top-level class implementing the interface, or an anonymous class implementing the interface, or a lambda, register it into the registry, and finally invoke it:

Registry registry = new Registry();
registry.registerCallback("hello", new Callback() {
    @Override 
    void call(String message) {
        System.out.println("Hello, here's your message: " + message);
    }
});

registry.invokeCallback("hello", "first message");