[英]spring batch using spring boot: Read arguments from config or command line and use them in job
[英]spring batch read jobParameters from command line and use it in job config
基於建議的編輯:為簡潔起見,我將刪除舊代碼和長部分並重新表述問題。
我正在嘗試從命令行獲取日期和配置信息來構建應用程序(Spring boot + Spring Batch)。 根據建議,我可以使用應用程序屬性嗎? 主要目的是使用相同的作業(作業的任務)從不同的主機/時間等下載不同的文件。因此,屬性文件可以提供用於下載的信息,編譯的 jar 應該讀取信息並執行其任務。
主要入口點。
@SpringBootApplication
public class CoreApplication implements ApplicationRunner {
@Autowired
JobLauncher jobLauncher;
@Autowired
Job processJob;
@Value("${rundate}")
private String run_date;
private static final Logger logger = LoggerFactory.getLogger(CoreApplication.class);
public static void main(String[] args) {
SpringApplication.run(CoreApplication.class, args);
}
@Override
public void run(ApplicationArguments args) throws Exception {
JobParameters jobParameters = new JobParametersBuilder()
.addLong("JobID", System.currentTimeMillis())
.addString("RunDate", run_date)
.toJobParameters();
try {
jobLauncher.run(processJob, jobParameters);
} catch (Exception e) {
logger.error("Exception while running a batch job {}", e.getMessage());
}
}
}
我重新排列了代碼,以使用application.properties文件中的服務器、用戶等的值。 請讓我知道注入屬性的方法是否錯誤。
application.properties 文件:
spring.datasource.url=jdbc:postgresql://dbhost:1000/db
spring.datasource.username=username
spring.datasource.password=password
spring.datasource.platform=postgresql
spring.batch.job.enabled=false
local.directory="/my/local/path/"
file.name="file_name_20200601.csv"
remote.directory="/remote/ftp/location"
remote.host="remotehost"
remote.port=22
remote.user="remoteuser"
private.key.location="/key/file/location"
我的批量配置:
@Configuration
@EnableBatchProcessing
@EnableIntegration
@EnableAutoConfiguration
public class BatchConfiguration {
private Logger logger = LoggerFactory.getLogger(BatchConfiguration.class);
@Autowired
public JobBuilderFactory jobBuilderFactory;
@Autowired
public StepBuilderFactory stepBuilderFactory;
@Bean
public Job ftpJob() {
return jobBuilderFactory.get("FTP Job")
.incrementer(new RunIdIncrementer())
.start(getFilesFromFTPServer())
.build();
}
@Bean
public Step getFilesFromFTPServer() {
return stepBuilderFactory.get("Get file from server")
.tasklet(new RemoteFileInboundTasklet())
.build();
}
}
我的小任務:
公共 class RemoteFileInboundTasklet 實現 Tasklet {
private Logger logger = LoggerFactory.getLogger(RemoteFileInboundTasklet.class);
@Value("${file.name}")
private String fileNamePattern;
private String clientName;
private boolean deleteLocalFiles = true;
private boolean retryIfNotFound = false;
@Value("${local.directory}")
private String local_directory_value;
private File localDirectory;
private int downloadFileAttempts = 12;
private long retryIntervalMilliseconds = 300000;
@Value("${remote.directory}")
private String remoteDirectory;
@Value("${remote.host}")
private String remoteHost;
@Value("${remote.user}")
private String remoteUser;
@Value("${remote.port}")
private int remotePort;
@Value("${private.key.location}")
private String private_key_file;
public SessionFactory<ChannelSftp.LsEntry> clientSessionFactory() {
DefaultSftpSessionFactory ftpSessionFactory = new DefaultSftpSessionFactory();
ftpSessionFactory.setHost(remoteHost);
ftpSessionFactory.setPort(remotePort);
ftpSessionFactory.setUser(remoteUser);
ftpSessionFactory.setPrivateKey(new FileSystemResource(private_key_file));
ftpSessionFactory.setAllowUnknownKeys(true);
return ftpSessionFactory;
}
private SessionFactory sessionFactory = clientSessionFactory();
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer sftpInboundFileSynchronizer = new SftpInboundFileSynchronizer(sessionFactory);
sftpInboundFileSynchronizer.setDeleteRemoteFiles(false);
sftpInboundFileSynchronizer.setRemoteDirectory(remoteDirectory);
return sftpInboundFileSynchronizer;
}
private SftpInboundFileSynchronizer ftpInboundFileSynchronizer = sftpInboundFileSynchronizer();
private SftpInboundFileSynchronizingMessageSource sftpInboundFileSynchronizingMessageSource;
public boolean isDeleteLocalFiles() {
return deleteLocalFiles;
}
public void setDeleteLocalFiles(boolean deleteLocalFiles) {
this.deleteLocalFiles = deleteLocalFiles;
}
public SftpInboundFileSynchronizer getFtpInboundFileSynchronizer() {
return ftpInboundFileSynchronizer;
}
public void setFtpInboundFileSynchronizer(SftpInboundFileSynchronizer ftpInboundFileSynchronizer) {
this.ftpInboundFileSynchronizer = ftpInboundFileSynchronizer;
}
public SessionFactory getSessionFactory() {
return sessionFactory;
}
public void setSessionFactory(SessionFactory sessionFactory) {
this.sessionFactory = sessionFactory;
}
public SftpInboundFileSynchronizingMessageSource getSftpInboundFileSynchronizingMessageSource() {
return sftpInboundFileSynchronizingMessageSource;
}
public void setSftpInboundFileSynchronizingMessageSource(SftpInboundFileSynchronizingMessageSource sftpInboundFileSynchronizingMessageSource) {
this.sftpInboundFileSynchronizingMessageSource = sftpInboundFileSynchronizingMessageSource;
}
public String getRemoteDirectory() {
return remoteDirectory;
}
public void setRemoteDirectory(String remoteDirectory) {
this.remoteDirectory = remoteDirectory;
}
private SFTPGateway sftpGateway;
@ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler clientMessageHandler() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(clientSessionFactory(), "mget", "payload");
sftpOutboundGateway.setAutoCreateLocalDirectory(true);
sftpOutboundGateway.setLocalDirectory(new File(local_directory_value));
sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
sftpOutboundGateway.setFilter(new AcceptOnceFileListFilter<>());
return sftpOutboundGateway;
}
private void deleteLocalFiles()
{
if (deleteLocalFiles)
{
localDirectory = new File(local_directory_value);
SimplePatternFileListFilter filter = new SimplePatternFileListFilter(fileNamePattern);
List<File> matchingFiles = filter.filterFiles(localDirectory.listFiles());
if (CollectionUtils.isNotEmpty(matchingFiles))
{
for (File file : matchingFiles)
{
FileUtils.deleteQuietly(file);
}
}
}
}
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
deleteLocalFiles();
ftpInboundFileSynchronizer.synchronizeToLocalDirectory(localDirectory);
if (retryIfNotFound) {
SimplePatternFileListFilter filter = new SimplePatternFileListFilter(fileNamePattern);
int attemptCount = 1;
while (filter.filterFiles(localDirectory.listFiles()).size() == 0 && attemptCount <= downloadFileAttempts) {
logger.info("File(s) matching " + fileNamePattern + " not found on remote site. Attempt " + attemptCount + " out of " + downloadFileAttempts);
Thread.sleep(retryIntervalMilliseconds);
ftpInboundFileSynchronizer.synchronizeToLocalDirectory(localDirectory);
attemptCount++;
}
if (attemptCount >= downloadFileAttempts && filter.filterFiles(localDirectory.listFiles()).size() == 0) {
throw new FileNotFoundException("Could not find remote file(s) matching " + fileNamePattern + " after " + downloadFileAttempts + " attempts.");
}
}
return RepeatStatus.FINISHED;
}
public String getFileNamePattern() {
return fileNamePattern;
}
public void setFileNamePattern(String fileNamePattern) {
this.fileNamePattern = fileNamePattern;
}
public String getClientName() {
return clientName;
}
public void setClientName(String clientName) {
this.clientName = clientName;
}
public boolean isRetryIfNotFound() {
return retryIfNotFound;
}
public void setRetryIfNotFound(boolean retryIfNotFound) {
this.retryIfNotFound = retryIfNotFound;
}
public File getLocalDirectory() {
return localDirectory;
}
public void setLocalDirectory(File localDirectory) {
this.localDirectory = localDirectory;
}
public int getDownloadFileAttempts() {
return downloadFileAttempts;
}
public void setDownloadFileAttempts(int downloadFileAttempts) {
this.downloadFileAttempts = downloadFileAttempts;
}
public long getRetryIntervalMilliseconds() {
return retryIntervalMilliseconds;
}
public void setRetryIntervalMilliseconds(long retryIntervalMilliseconds) {
this.retryIntervalMilliseconds = retryIntervalMilliseconds;
}
}
我的理解(如果有錯誤請在此處更正)application.properties 文件屬性可以注入到 tasklet 中(如上)。 然后我嘗試構建 package。
mvn clean package
我收到以下錯誤:
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.batch.core.Step]: Factory method 'getFilesFromFTPServer' threw exception; nested exception is java.lang.IllegalArgumentException: Path must not be null
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
... 122 common frames omitted
Caused by: java.lang.IllegalArgumentException: Path must not be null
at org.springframework.util.Assert.notNull(Assert.java:198) ~[spring-core-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.core.io.FileSystemResource.<init>(FileSystemResource.java:80) ~[spring-core-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at com.my.batch.core.tasklet.RemoteFileInboundTasklet.clientSessionFactory(RemoteFileInboundTasklet.java:78) ~[classes/:na]
at com.my.batch.core.tasklet.RemoteFileInboundTasklet.<init>(RemoteFileInboundTasklet.java:83) ~[classes/:na]
at com.my.batch.core.BatchConfiguration.getFilesFromFTPServer(BatchConfiguration.java:71) ~[classes/:na]
at com.my.batch.core.BatchConfiguration$$EnhancerBySpringCGLIB$$17d8a6d9.CGLIB$getFilesFromFTPServer$1(<generated>) ~[classes/:na]
代碼中的行是:
ftpSessionFactory.setPrivateKey(new FileSystemResource(private_key_file));
通過 BatchConfiguration.java -> getFilesFromFTPServer 調用。
這意味着我來自 applcation.properties 的值沒有通過? 我需要做哪些改變?
而且,在編譯或構建 jar 時,為什么要檢查變量的值?
新編輯:
我試圖在 Configuration 中將我的 tasklet 聲明為 bean 並再次構建 package。 但是,它給出了同樣的錯誤。
更改后我的 application.properties 文件:
spring.datasource.url=jdbc:postgresql://dbhost:1000/db
spring.datasource.username=username
spring.datasource.password=password
spring.datasource.platform=postgresql
spring.batch.job.enabled=false
local.directory=/my/local/path/
file.name=file_name_20200601.csv
remote.directory=/remote/ftp/location
remote.host=remotehost
remote.port=22
remote.user=remoteuser
private.key.location=/key/file/location
tasklet 沒有變化。
更改配置:
@Configuration
@EnableBatchProcessing
@EnableIntegration
@EnableAutoConfiguration
public class BatchConfiguration {
private Logger logger = LoggerFactory.getLogger(BatchConfiguration.class);
@Autowired
public JobBuilderFactory jobBuilderFactory;
@Autowired
public StepBuilderFactory stepBuilderFactory;
@Bean
public RemoteFileInboundTasklet remoteFileInboundTasklet() {
return new RemoteFileInboundTasklet();
}
@Bean
public Job ftpJob() {
return jobBuilderFactory.get("FTP Job")
.incrementer(new RunIdIncrementer())
.start(getFilesFromFTPServer())
.build();
}
@Bean
public Step getFilesFromFTPServer() {
return stepBuilderFactory.get("Get file from server")
.tasklet(remoteFileInboundTasklet())
.build();
}
}
當我嘗試構建包(mvn clean package)時,我仍然得到同樣的錯誤。
路徑不得為 null。
它無法讀取屬性。 知道有什么問題嗎?
基於不同方法的編輯:
我試圖進一步了解如何使用配置並找到以下方法來使用 @ConfigurationProperties 注釋( 如何訪問 Spring Boot 中的 application.properties 文件中定義的值)
我創建了一個新的 ftp 配置 class:
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@ConfigurationProperties(prefix = "ftp")
@Configuration("coreFtpProperties")
public class CoreFtp {
private String host;
private String port;
private String user;
private String passwordKey;
private String localDirectory;
private String remoteDirectory;
private String fileName;
public String getHost() {
return host;
}
public void setHost(String host) {
this.host = host;
}
public String getPort() {
return port;
}
public void setPort(String port) {
this.port = port;
}
public String getUser() {
return user;
}
public void setUser(String user) {
this.user = user;
}
public String getPasswordKey() {
return passwordKey;
}
public void setPasswordKey(String passwordKey) {
this.passwordKey = passwordKey;
}
public String getLocalDirectory() {
return localDirectory;
}
public void setLocalDirectory(String localDirectory) {
this.localDirectory = localDirectory;
}
public String getRemoteDirectory() {
return remoteDirectory;
}
public void setRemoteDirectory(String remoteDirectory) {
this.remoteDirectory = remoteDirectory;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
}
對 application.properties 文件的小改動:
spring.datasource.url=jdbc:postgresql://dbhost:1000/db
spring.datasource.username=username
spring.datasource.password=password
spring.datasource.platform=postgresql
spring.batch.job.enabled=false
ftp.local_directory=/my/local/path/
ftp.file_name=file_name_20200601.csv
ftp.remote_directory=/remote/ftp/location
ftp.host=remotehost
ftp.port=22
ftp.user=remoteuser
ftp.password_key=/key/file/location
在我的批處理配置中,我進行了以下更改:
@Configuration
@EnableBatchProcessing
@EnableIntegration
public class BatchConfiguration {
private Logger logger = LoggerFactory.getLogger(BatchConfiguration.class);
@Autowired
public JobBuilderFactory jobBuilderFactory;
@Autowired
public StepBuilderFactory stepBuilderFactory;
@Autowired
private CoreFtp coreFtpProperties;
@Bean
public RemoteFileInboundTasklet remoteFileInboundTasklet() {
RemoteFileInboundTasklet ftpTasklet = new RemoteFileInboundTasklet();
ftpTasklet.setRetryIfNotFound(true);
ftpTasklet.setDownloadFileAttempts(3);
ftpTasklet.setRetryIntervalMilliseconds(10000);
ftpTasklet.setFileNamePattern(coreFtpProperties.getFileName());
ftpTasklet.setRemoteDirectory(coreFtpProperties.getRemoteDirectory());
ftpTasklet.setLocalDirectory(new File(coreFtpProperties.getLocalDirectory()));
ftpTasklet.setSessionFactory(clientSessionFactory());
ftpTasklet.setFtpInboundFileSynchronizer(sftpInboundFileSynchronizer());
ftpTasklet.setSftpInboundFileSynchronizingMessageSource(new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer()));
return ftpTasklet;
}
@Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer sftpInboundFileSynchronizer = new SftpInboundFileSynchronizer(clientSessionFactory());
sftpInboundFileSynchronizer.setDeleteRemoteFiles(false);
sftpInboundFileSynchronizer.setRemoteDirectory(coreFtpProperties.getRemoteDirectory());
return sftpInboundFileSynchronizer;
}
@Bean(name = "clientSessionFactory")
public SessionFactory<LsEntry> clientSessionFactory() {
DefaultSftpSessionFactory ftpSessionFactory = new DefaultSftpSessionFactory();
ftpSessionFactory.setHost(coreFtpProperties.getHost());
ftpSessionFactory.setPort(Integer.parseInt(coreFtpProperties.getPort()));
ftpSessionFactory.setUser(coreFtpProperties.getUser());
ftpSessionFactory.setPrivateKey(new FileSystemResource(coreFtpProperties.getPasswordKey()));
ftpSessionFactory.setPassword("");
ftpSessionFactory.setAllowUnknownKeys(true);
return ftpSessionFactory;
}
@Bean
@ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler clientMessageHandler() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(clientSessionFactory(), "mget", "payload");
sftpOutboundGateway.setAutoCreateLocalDirectory(true);
sftpOutboundGateway.setLocalDirectory(new File(coreFtpProperties.getLocalDirectory()));
sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
sftpOutboundGateway.setFilter(new AcceptOnceFileListFilter<>());
return sftpOutboundGateway;
}
@Bean
public Job ftpJob() {
return jobBuilderFactory.get("FTP Job")
.incrementer(new RunIdIncrementer())
.start(getFilesFromFTPServer())
.build();
}
@Bean
public Step getFilesFromFTPServer() {
return stepBuilderFactory.get("Get file from server")
.tasklet(remoteFileInboundTasklet())
.build();
}
}
因此,因此我的 Tasklet 更改為:
public class RemoteFileInboundTasklet implements Tasklet {
private Logger logger = LoggerFactory.getLogger(RemoteFileInboundTasklet.class);
private String fileNamePattern;
private String clientName;
private boolean deleteLocalFiles = true;
private boolean retryIfNotFound = false;
private File localDirectory;
private int downloadFileAttempts = 12;
private long retryIntervalMilliseconds = 300000;
private String remoteDirectory;
private SessionFactory sessionFactory;
private SftpInboundFileSynchronizer ftpInboundFileSynchronizer;
private SftpInboundFileSynchronizingMessageSource sftpInboundFileSynchronizingMessageSource;
public boolean isDeleteLocalFiles() {
return deleteLocalFiles;
}
public void setDeleteLocalFiles(boolean deleteLocalFiles) {
this.deleteLocalFiles = deleteLocalFiles;
}
public SftpInboundFileSynchronizer getFtpInboundFileSynchronizer() {
return ftpInboundFileSynchronizer;
}
public void setFtpInboundFileSynchronizer(SftpInboundFileSynchronizer ftpInboundFileSynchronizer) {
this.ftpInboundFileSynchronizer = ftpInboundFileSynchronizer;
}
public SessionFactory getSessionFactory() {
return sessionFactory;
}
public void setSessionFactory(SessionFactory sessionFactory) {
this.sessionFactory = sessionFactory;
}
public SftpInboundFileSynchronizingMessageSource getSftpInboundFileSynchronizingMessageSource() {
return sftpInboundFileSynchronizingMessageSource;
}
public void setSftpInboundFileSynchronizingMessageSource(SftpInboundFileSynchronizingMessageSource sftpInboundFileSynchronizingMessageSource) {
this.sftpInboundFileSynchronizingMessageSource = sftpInboundFileSynchronizingMessageSource;
}
public String getRemoteDirectory() {
return remoteDirectory;
}
public void setRemoteDirectory(String remoteDirectory) {
this.remoteDirectory = remoteDirectory;
}
private SFTPGateway sftpGateway;
private void deleteLocalFiles()
{
if (deleteLocalFiles)
{
SimplePatternFileListFilter filter = new SimplePatternFileListFilter(fileNamePattern);
List<File> matchingFiles = filter.filterFiles(localDirectory.listFiles());
if (CollectionUtils.isNotEmpty(matchingFiles))
{
for (File file : matchingFiles)
{
FileUtils.deleteQuietly(file);
}
}
}
}
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
deleteLocalFiles();
ftpInboundFileSynchronizer.synchronizeToLocalDirectory(localDirectory);
if (retryIfNotFound) {
SimplePatternFileListFilter filter = new SimplePatternFileListFilter(fileNamePattern);
int attemptCount = 1;
while (filter.filterFiles(localDirectory.listFiles()).size() == 0 && attemptCount <= downloadFileAttempts) {
logger.info("File(s) matching " + fileNamePattern + " not found on remote site. Attempt " + attemptCount + " out of " + downloadFileAttempts);
Thread.sleep(retryIntervalMilliseconds);
ftpInboundFileSynchronizer.synchronizeToLocalDirectory(localDirectory);
attemptCount++;
}
if (attemptCount >= downloadFileAttempts && filter.filterFiles(localDirectory.listFiles()).size() == 0) {
throw new FileNotFoundException("Could not find remote file(s) matching " + fileNamePattern + " after " + downloadFileAttempts + " attempts.");
}
}
return RepeatStatus.FINISHED;
}
}
基於上述更改,我能夠編譯代碼並創建必要的 Jar,並使用 jar 運行代碼。
您正在聲明一個 bean jobExecutionListener()
在其中創建new FileSystemResource(config_file_path);
. config_file_path
是從作業參數@Value("#{jobParameters['ConfigFilePath']}")
注入的,這些參數在配置時不可用,但僅在運行作業/步驟時才可用。 這稱為后期綁定。
因此,在您的情況下,當 Spring 嘗試創建 bean jobExecutionListener()
時,它會嘗試注入config_file_path
但當時是 null (此時 Z38008DD81C2F4D7985ECF1 bean 並沒有運行以配置應用程序)作業只是創建應用程序因此尚未執行beforeJob
方法。 這就是你有NullPointerException
的原因。 在 jobExecutionListener jobExecutionListener()
bean 上添加@JobScope
應該可以解決問題,但我不建議這樣做。 原因是您試圖以錯誤的方式和錯誤的位置配置某些屬性,因此我會修復該設計,而不是通過添加注釋來解決問題。
作業參數用於業務參數而非技術細節。 在您的情況下, runDate
是作業參數的不錯選擇,但不是ConfigFilePath
。 而且,既然你用的是Spring,為什么要注入文件路徑然后做properties = PropertiesLoaderUtils.loadProperties(resource);
和Integer.parseInt(properties.getProperty("remote.port"));
? 如果告訴它在需要的地方注入屬性,Spring 將為您執行此操作。
我將刪除此config_file_path
作業參數以及作業偵聽器,並直接在remoteFileInboundTasklet
中注入屬性,即盡可能靠近需要這些屬性的位置。
編輯:添加代碼示例
您能否幫助了解我在哪里可以將 tasklet 聲明為 bean?
在您的步驟getFilesFromFTPServer
中,您正在手動創建 tasklet,因此不執行依賴注入。 您需要將 tasklet 聲明為 Spring bean 才能使其工作,例如:
@Bean
public Tasklet myTasklet() {
return new RemoteFileInboundTasklet()
}
@Bean
public Step getFilesFromFTPServer() {
return stepBuilderFactory.get("Get file from server")
.tasklet(myTasklet())
.build();
}
您需要將getFilesFromFTPServer
bean 更改為 JobScope 並從那里讀取所有作業運行時參數。
@Bean
@JobScope
public Step getFilesFromFTPServer() {
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.