[英]Spring batch remote partitioning to multiple application instances
I am working on remote partitioning using spring batch.. I have multiple instances of spring batch application running.我正在使用 spring 批处理进行远程分区。我有多个 spring 批处理应用程序实例正在运行。 As part of use case I need to process all records present in database.作为用例的一部分,我需要处理数据库中存在的所有记录。 So I am assigning instances to every records.. So each instance of batch application can process instance specific records.所以我为每个记录分配实例。所以批处理应用程序的每个实例都可以处理实例特定的记录。
eg例如
Records1 -: instance1
Records2 -: instance1
Records3 -: instance2
Records4 -: instance2
.... so on
I know we can use Kafka or JMS instead of partitioning explicitly.我知道我们可以使用 Kafka 或 JMS 而不是显式分区。 In my use case I don't want to use any messaging middleware.在我的用例中,我不想使用任何消息中间件。 So after assigning instances to each records.因此,在将实例分配给每个记录之后。 I want to invoke both instances.我想调用这两个实例。 How can I invoke both instances?如何调用这两个实例?
You still need a messaging middleware to implement remote partitioning.你仍然需要一个消息中间件来实现远程分区。 This is required for the manager step to send StepExecutionRequest
s to worker steps.这是管理步骤将StepExecutionRequest
发送到工作步骤所必需的。
If you really want to avoid having a messaging middleware, you can create a job instance per record set.如果您真的想避免使用消息传递中间件,您可以为每个记录集创建一个作业实例。 According to your example, this could be designed as follows:根据您的示例,这可以设计如下:
Those job instances can be executed locally in different threads of the same JVM, or locally in different JVMs on the same machine, or remotely on different machines in a cluster.这些作业实例可以在同一 JVM 的不同线程中本地执行,也可以在同一机器上的不同 JVM 中本地执行,或者在集群中的不同机器上远程执行。 The degree of parallelism depends only on the resources available in your cluster.并行度仅取决于集群中可用的资源。 You can have, on a single machine, several JVMs where each JVM is running several job instances in parallel.您可以在一台机器上拥有多个 JVM,其中每个 JVM 并行运行多个作业实例。 You could then duplicate that setup on different machines and achieve a high level of parallelism for you requirement.然后,您可以在不同的机器上复制该设置,并根据您的要求实现高水平的并行性。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.