基于 Spring Boot 的应用程序中 Spark 和 ObjectMapper 的序列化问题

Question

I'm using Spark and there's one of my Spring Boot-based application beans:

@Component
@RequiredArgsConstructor
public class SomeService implements FlatMapFunction<T, K> {

  private final ObjectMapper mapper;
  
}

ObjectMapper here is the standard one taken from application context. The problem is that the app fails with org.apache.spark.SparkException: Task not serializable . Here's serialization stack:

Caused by: java.io.NotSerializableException: org.springframework.http.converter.json.SpringHandlerInstantiator
Serialization stack:
- object not serializable (class: org.springframework.http.converter.json.SpringHandlerInstantiator, value: org.springframework.http.converter.json.SpringHandlerInstantiator@6e4912db)
- field (class: com.fasterxml.jackson.databind.cfg.BaseSettings, name: _handlerInstantiator, type: class com.fasterxml.jackson.databind.cfg.HandlerInstantiator)
- object (class com.fasterxml.jackson.databind.cfg.BaseSettings, com.fasterxml.jackson.databind.cfg.BaseSettings@155616d8)
- field (class: com.fasterxml.jackson.databind.cfg.MapperConfig, name: _base, type: class com.fasterxml.jackson.databind.cfg.BaseSettings)
- object (class com.fasterxml.jackson.databind.DeserializationConfig, com.fasterxml.jackson.databind.DeserializationConfig@66e72ca2)
- field (class: com.fasterxml.jackson.databind.ObjectMapper, name: _deserializationConfig, type: class com.fasterxml.jackson.databind.DeserializationConfig)
- object (class com.fasterxml.jackson.databind.ObjectMapper, com.fasterxml.jackson.databind.ObjectMapper@433ef204)
- field (class: com.smth.SomeService, name: mapper, type: class com.fasterxml.jackson.databind.ObjectMapper)

So the problem is about non-serializable SpringHandlerInstantiator .

This far I work this around by assigning mapper field in constructor manually:

public SomeService() {
  this.mapper = new ObjectMapper();
}

Is there a way to somehow solve this properly, ie relying on Spring's DI?

I use Spring Boot 2.6.7 and Spark 2.11.

Answer 1

Try to configure ObjectMapper bean to use a serializable HandlerInstantiator. In the following example MyHandlerInstantiator is a custom implementation of HandlerInstantiator which is serializable (eg it can be started with minimal patching the code of SpringHandlerInstantiator ). This should allow SomeService class to be serializable as well and compatible for usage in a distributed environment like Apache Spark.

@Configuration
public class MyConfiguration {
  
  @Bean
  public ObjectMapper objectMapper() {
    ObjectMapper mapper = new ObjectMapper();
    mapper.setHandlerInstantiator(new MyHandlerInstantiator());
    return mapper;
  }
}