[英]Integration - Apache Flink + Spring Boot
I'm testing the integration between Apache Flink and Spring Boot, to run them on IDE is fine, but when I tried to run on Apache Flink Cluster I had one Exception related to ClassLoader. 我正在测试Apache Flink和Spring Boot之间的集成,在IDE上运行它们很好,但是当我尝试在Apache Flink Cluster上运行时,我遇到了一个与ClassLoader相关的异常。
The classes are really simple: 这些课很简单:
BootFlinkApplication BootFlinkApplication
@SpringBootApplication
@ComponentScan("com.example.demo")
public class BootFlinkApplication {
public static void main(String[] args) {
System.out.println("some test");
SpringApplication.run(BootFlinkApplication.class, args);
}
}
FlinkTest FlinkTest
@Service
public class FlinkTest {
@PostConstruct
public void init() {
StreamExecutionEnvironment see = StreamExecutionEnvironment.getExecutionEnvironment();
see.fromElements(1, 2, 3, 4)
.filter(new RemoveNumber3Filter()).print();
try {
see.execute();
} catch (Exception e) {
System.out.println("Error executing flink job: " + e.getMessage());
}
}
}
RemoveNumber3Filter RemoveNumber3Filter
public class RemoveNumber3Filter implements FilterFunction<Integer> {
@Override
public boolean filter(Integer i) throws Exception {
return i != 3;
}
}
Exception: 例外:
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:897)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:840)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:840)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.example.demo.RemoveNumber3Filter
ClassLoader info: URL ClassLoader:
file: '/tmp/blobStore-850f3189-807e-4f8d-a8a6-3bd3c1bd76b4/job_eb93b239080b4d4e09f10f1e3605744d/blob_p-5fd56f3348976c0d333d680fde4a79573c21cd40-48ac0995eee11f38ce3ff4f890102af8' (valid JAR)
Class not resolvable through given classloader.
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:232)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:355)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:282)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:126)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:231)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
at java.lang.Thread.run(Thread.java:748)
You probably use the Spring Boot Maven plugin ( https://docs.spring.io/spring-boot/docs/current/reference/html/build-tool-plugins-maven-plugin.html ) to repackage your Jar to generate an executable jar, however, it uses a custom boot layout which is not supported by Apache Flink's internal class loader. 您可能使用Spring Boot Maven插件( https://docs.spring.io/spring-boot/docs/current/reference/html/build-tool-plugins-maven-plugin.html )重新打包您的Jar以生成但是,它使用了一个自定义引导布局,Apache Flink的内部类加载器不支持它。 There should be an original jar file next to the one you try to deploy (.jar.original) which you could use for deployment on the Flink cluster. 您尝试部署的文件(.jar.original)旁边应该有一个原始jar文件,您可以将其用于Flink集群上的部署。
As an alternative, you can use a different way of generating a Jar with all your dependencies, such as maven-shade ( https://maven.apache.org/plugins/maven-shade-plugin/ ) 作为替代方案,您可以使用不同的方式生成具有所有依赖项的Jar,例如maven-shade( https://maven.apache.org/plugins/maven-shade-plugin/ )
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.