如何在 spring-boot 多模块 Java 11 项目中添加 Spark 依赖

Question

Whenever I am adding a module-info.java in my multi-module project I cannot import my Spark dependencies - everything else seems to be working

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.12</artifactId>
    <version>3.0.0-preview2</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>3.0.0-preview2</version>
</dependency>

IntelliJ tries to readd Maven Dependency without any result.

My module-info looks like:

module common {
    exports [...] 
    requires lombok;
    requires spring.data.jpa;
    requires spring.data.commons;
    requires org.apache.commons.lang3;
    requires spring.context;
    requires spring.web;
    requires spring.security.core;
    requires com.google.common;
    requires org.json;
    requires spring.core;
    requires spring.beans;
    requires com.fasterxml.jackson.core;
    requires com.fasterxml.jackson.databind;
    requires spring.jcl;
    requires spring.webmvc;
    requires mongo.java.driver;
    requires org.hibernate.orm.core;
    requires com.fasterxml.jackson.dataformat.csv;
    requires java.sql;
}

It is not possible to add org.apache.* in my module-info.java either.

Is it possible that Spark is not ready for Jigsaw modules and Java 9+?

Answer 1

Is it possible that spark is not ready for Jigsaw modules and Java 9+?

It does hold true for spark . Two straight reasons that I can vouch for are:

1. They do not have an entry for

   Automatic-Module-Name: <module-name>

   in the artifact's MANIFEST.MF file.

2. If you try describing their artifacts using the jar tool

   jar --describe-module --file=<complete-path>/spark-core_2.12-3.0.0-preview2.jar

   This would fail to derive the module descriptor for a similar reason as mentioned in this answer.

---

Few resources that might be useful once you reach here:

• The reason why deriving automatic module name fails for spark artifacts
• A way to update a jar manually with the MANIFEST entry
• Spark's progress to Build and Run on JDK-11