使用 DeployerPartitionHandler 在通过 spring 云任务运行的 spring 批处理中获取空指针异常

Question

我正在使用 DeployerPartitionHandler（本地变体）来分区我的 Spring 批处理作业。 当我运行我的工作时，我在工作程序的启动步骤中收到一个空指针异常，如下所示

at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorkers(DeployerPartitionHandler.java:313)
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.handle(DeployerPartitionHandler.java:302)
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:106)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:410)
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:136)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:319)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:147)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy248.run(Unknown Source)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:207)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:181)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:168)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:163)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:781)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:765)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:319)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1204).

作业开始时我看到以下错误消息

错误 [oscloud.task.listener.TaskLifecycleListener ] 对于尚未启动的任务，已收到结束任务的事件。 [主要的] []

但它随后在 TaskRepository 中创建一个条目，如下所示并继续创建分区

DEBUG [osctask.repository.support.SimpleTaskRepository] ​​创建：TaskExecution{executionId=0, parentExecutionId=null, exitCode=null, taskName='MYTASK', startTime=Tue Nov 24 17:14:48 IST 2020, endTime=null, exitMessage ='null', externalExecutionId='null', errorMessage='null', arguments=[--spring.profiles.active=local, --spring.output.ansi.enabled=always]} [main] []

DEBUG [osctbatch.partition.DeployerPartitionHandler] 返回了 3 个分区 [main] []

此外，我在 TaskRepository 中看到另一个条目，但这次它在大多数列中使用空值DEBUG [osctask.repository.support.SimpleTaskRepository] ​​创建：TaskExecution{executionId=65, parentExecutionId=null, exitCode=null, taskName='null ', startTime=null, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[]} [main] []

我认为由于 spring 批处理是一项单一任务，我应该期望 TaskRepository 中只有一个条目，但无法弄清楚为什么要创建第二个条目我正在使用 Postgres 并遵循示例代码中提到的相同步骤，但无法计算出问题https://github.com/spring-cloud/spring-cloud-task/tree/master/spring-cloud-task-samples/partitioned-batch-job

无论如何，下面是代码以供参考

@Bean
@Profile("!worker")
public PartitionHandler partitionHandler(TaskLauncher taskLauncher, JobExplorer jobExplorer,
        TaskRepository taskRepository) throws Exception {
    

    Resource resource = this.resourceLoader.getResource("maven://XXX:YYYY:0.0.2-SNAPSHOT");

    DeployerPartitionHandler partitionHandler =
                new DeployerPartitionHandler(taskLauncher, jobExplorer, resource, "workerStep", taskRepository);
      
      
    List<String> commandLineArgs = new ArrayList<>(3);
    commandLineArgs.add("--spring.profiles.active=worker");
    commandLineArgs.add("--spring.cloud.task.initialize-enabled=false");
    commandLineArgs.add("--spring.batch.initializer.enabled=false");
    partitionHandler.setCommandLineArgsProvider(new PassThroughCommandLineArgsProvider(commandLineArgs));
    partitionHandler.setEnvironmentVariablesProvider(new SimpleEnvironmentVariablesProvider(this.environment));
    partitionHandler.setMaxWorkers(2);
    partitionHandler.setApplicationName("MYTASK");

    return partitionHandler;
}


@Bean
@Profile("!worker")
public Job partitionedJob(PartitionHandler partitionHandler) throws Exception {
    
    Random random = new Random();
    return this.jobBuilderFactory.get("partitionedJob" + random.nextInt())
        .start(step1(partitionHandler))
        .build();
}

@Bean
@Profile("!worker")
public Step step1(PartitionHandler partitionHandler) throws Exception {

    return this.stepBuilderFactory.get("step1")
        .partitioner(workerStep().getName(), partitioner())         
        .step(workerStep())
        .partitionHandler(partitionHandler)
        .build();
}


@Bean
@Profile("!worker")
public Partitioner partitioner() {
    return new Partitioner() {
        @Override
        public Map<String, ExecutionContext> partition(int gridSize) {
            System.out.println("In partitioner");
            Map<String, ExecutionContext> partitions = new HashMap<>(gridSize);

            for (int i = 0; i < 3; i++) {
                ExecutionContext context1 = new ExecutionContext();
                context1.put("partitionNumber", i);

                partitions.put("partition" + i, context1);
            }
            
            return partitions;
        }
    };
}

@Bean
@Profile("worker")
public DeployerStepExecutionHandler stepExecutionHandler(JobExplorer jobExplorer) {
    
    return new DeployerStepExecutionHandler(this.context, jobExplorer, this.jobRepository);
}

@Bean

public Step workerStep() {

    return this.stepBuilderFactory.get("workerStep")
        .tasklet(workerTasklet(null))
    
        .build();
}

@Bean
@StepScope
public Tasklet workerTasklet(
    final @Value("#{stepExecutionContext['partitionNumber']}") Integer partitionNumber) {

    return new Tasklet() {
        @Override
        public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {

            return RepeatStatus.FINISHED;
        }
    };
}

Answer 1

因为代码看起来有点不完整，所以不能100%确定。

但是在添加后我能够解决类似的问题

1. @EnableTask

在定义此作业配置文件的配置类上

2. 并确保partioniner 给出的值小于或等于maxworker（分区器似乎为for 循环提供了3 个网格，但maxworker 仅为2）

关于任务执行编号，以下是我将 TaskRepository 也映射到 DB（而不是内存中）后发现的内容 可以在此处参考代码/信息如何将 TaskRepository 与实际 DB 映射而不是默认内存https:/ /www.baeldung.com/spring-cloud-task

如果从新开始，假设在上面的代码中设置了 2 个工作器（分区器在 for 循环中添加了 3 个网格，但 MAXWORKER -> 2，因此根据我的理解将设置为 max）

TASK 1 将链接到 JOB 1 TASK 2 -> worker #1 TASK 3 -> worker #2

所以似乎spring cloud task也会自动为每个launch worker创建一个任务来进行分区（或者它是网格）。

任务 2 和任务 3 将在另一个表中有一个映射，以提及它们与任务 #1 相关联