Flink 无法从通过 PipelineOptions 添加的 JAR 加载 class

Question

I am developing a Java application which uses UDFs on Flink 1.14. I am using PipelineOptions.JARS config to add jar files containing UDF classes dynamically in the application code, However application fails to load UDF class from configured jar files with ClassNotFoundException .

I also tried PipelineOptions.CLASSPATHS , it fails with exact same error and stack trace.

The Same application jar works fine if submitted via Flink CLI using 'flink run' with the "-C" option to update classpath:

<FLINK_HOME>/bin/flink run --detached -C file:///path/to/udf.jar...

The problem seems to be that the classpath for the ClassLoader which codegen in table planner uses is not updated according to Configuration passed to the StreamExecutionEnvironment , and I am not sure how that can be done.

Here is how UDF jar file is added and UDF is registered:

final Configuration configuration = new Configuration();
configuration.set(PipelineOptions.JARS,Collections.singletonList("file:///path/to/udf.jar"));
StreamExecutionEnvironment streamEnv = StreamExecutionEnvironment.getExecutionEnvironment(configuration);
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(streamEnv);
...
Class udfClass = Class.forName("demo.MyUDF", ...);
tableEnv.createTemporarySystemFunction("MyUDF", udfClass);
...

Here is the error stack trace:

Exception in thread "main" java.lang.ClassNotFoundException: demo.MyUDF
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
    at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1886)
    at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1772)
    at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
    at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
    at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
    at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
    at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
    at org.apache.flink.table.planner.codegen.CodeGeneratorContext.addReusableObjectInternal(CodeGeneratorContext.scala:692)
    at org.apache.flink.table.planner.codegen.CodeGeneratorContext.addReusableFunction(CodeGeneratorContext.scala:714)
    at org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.generateFunctionAwareCall(BridgingFunctionGenUtil.scala:130)
    at org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.generateFunctionAwareCallWithDataType(BridgingFunctionGenUtil.scala:116)
    at org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.generateFunctionAwareCall(BridgingFunctionGenUtil.scala:73)
    at org.apache.flink.table.planner.codegen.calls.BridgingSqlFunctionCallGen.generate(BridgingSqlFunctionCallGen.scala:81)
    at org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:825)
    at org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:503)
    at org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:58)org.apache.flink.table.planner.delegation.StreamPlanner.translateToPlan(StreamPlanner.scala:70)
    at org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:185)
    at org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl.toStreamInternal(StreamTableEnvironmentImpl.java:437)
    at org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl.toStreamInternal(StreamTableEnvironmentImpl.java:432)
    at org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl.toDataStream(StreamTableEnvironmentImpl.java:356)
    ...

Answer 1

This error should come from the flink client, the JVM process of the client did not load file:///path/to/udf.jar

There are two ideas to try, the first is to put udf.jar under flink lib, and the second is that client jvm actively loads udf.jar

URLClassLoader classLoader = (URLClassLoader) Thread.currentThread().getContextClassLoader();
Class urlClass = URLClassLoader.class;
Method method = urlClass.getDeclaredMethod("addURL", new Class[]{URL.class});
method.setAccessible(true);
method.invoke(returnClassLoader, jarPath);