简体   繁体   English

SFTP Spring集成的文件轮询和过滤问题

[英]File Polling and Filtering Issues with SFTP Spring Integration

I am trying to pull remote files through SFTP Spring Integration and dump into my local drive, and kick-off a Spring Batch Job (Step-1 to read, process & write), followed by Step-2 where a Tasklet archives the local file downloaded to another location locally and deletes the downloaded file. 我试图通过SFTP Spring Integration提取远程文件并将其转储到本地驱动器,并启动Spring Batch作业(读取,处理和写入的步骤1),然后是步骤2,其中Tasklet将本地文件存档下载到本地的另一个位置并删除下载的文件。

Assuming the files being downloaded is "data.service", "data.maintenance" and "data.transaction". 假设要下载的文件是“ data.service”,“ data.maintenance”和“ data.transaction”。 I am only processing "data.service" but all 3 are archived and deleted. 我只处理“ data.service”,但所有3个都已归档并删除。

I want the above steps to happen at a specific interval assuming that a new set of files will be overwritten at the remote location. 我希望上述步骤在特定的时间间隔内进行,假设一组新的文件将在远程位置被覆盖。

Bean def 豆防御

<bean id="acceptAllFileListFilter" class="org.springframework.integration.file.filters.AcceptAllFileListFilter"/>
<bean id="acceptOnceFileListFilter" class="org.springframework.integration.file.filters.AcceptOnceFileListFilter"/>

conf CONF

<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
        channel="inboundFileChannel"
        session-factory="sftpSessionFactory"
        local-directory="file:${inbound.local.directory}"
        remote-directory="${inbound.remote.directory}"
        auto-create-local-directory="true"
        delete-remote-files="false"
        preserve-timestamp="true"
        filename-pattern="*.*"
        local-filter="acceptAllFileListFilter" >
    <int:poller max-messages-per-poll="-1" fixed-rate="60000" />
</int-sftp:inbound-channel-adapter>

What is happening now is... The above triggers this sequence and runs in an unending loop, without fetching the new files from remote. 现在发生的是...上面的操作触发了此序列,并以无休止的循环运行,而无需从远程获取新文件。 Details as below: 详细信息如下:

I start with an empty local folder, and remote server has all 3 files [Start the program] 我从一个空的本地文件夹开始,并且远程服务器具有所有3个文件[启动程序]

  1. Spring Integration program downloads all 3 files to local location Spring Integration程序将所有3个文件下载到本地位置
  2. Spring Batch Job Step-1 gets triggered Tasklet (Step-2) archives the Spring Batch Job Step-1被触发Tasklet(Step-2)将
  3. local files Tasklet (Step-2) deletes the local files Spring 本地文件Tasklet(第2步)删除Spring本地文件
  4. Integration program downloads all 3 files to local location (why???) 集成程序将所有3个文件下载到本地位置(为什么?)
  5. program keeps running Step-1 to Step-5 in an unending loop 程序在无休止的循环中继续运行步骤1到步骤5
  6. New file dropped on the remote server, same (1) to (5) no job triggered 新文件已删除到远程服务器上,与(1)到(5)相同,未触发任何作业

Tried replacing as below, but the program just runs once, ie, from (1) to (5) and nothing happens 尝试按以下方式替换,但是程序仅运行一次,即从(1)到(5),并且什么也没有

        local-filter="acceptOnceFileListFilter" >

Expected 预期

  1. Spring Integration program downloads all 3 files to local location Spring Integration程序将所有3个文件下载到本地位置
  2. Spring Batch Job Step-1 gets triggered Spring Batch Job Step-1被触发
  3. Tasklet (Step-2) archives the local files Tasklet(第2步)存档本地文件
  4. Tasklet (Step-2) deletes the local files Tasklet(第2步)删除本地文件
  5. Spring Integration program waits for new files to be dropped in the remote location (polls every 60000 ms) Spring Integration程序等待新文件放入远程位置(每60000毫秒轮询一次)
  6. New file dropped on the remote server 新文件已放置在远程服务器上
  7. Start with (1) through (6) 从(1)到(6)

Per suggestion on a different post I have a unique job parameter as below: 根据不同职位上的建议,我有一个唯一的工作参数,如下所示:

@Transformer
public JobLaunchRequest toRequest(Message<File> message) {
    JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
    jobParametersBuilder.addString(fileParameterName, message.getPayload().getAbsolutePath());
    jobParametersBuilder.addLong("currentTime", new Long(System.currentTimeMillis()));
    return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());
}

Tried with different combinations of filter, none served my purpose. 尝试使用不同的过滤器组合,但没有达到我的目的。 Any sample would be of great help, struggling with this since last week. 自上周以来一直在为此奋斗的任何样本都将大有帮助。

Here's my change per the suggestions: 根据建议,这是我的更改:

Adding a composite filter as below, but what would be the argument that will be passed to SftpPersistentAcceptOnceFileListFilter? 添加一个复合过滤器,如下所示,但是将传递给SftpPersistentAcceptOnceFileListFilter的参数是什么? I could not see any samples in the internet. 我在互联网上看不到任何样本。

<bean id="compositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
    <constructor-arg>
        <list>
            <bean class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter"/>
            <bean class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
            </bean>                     
        </list>
    </constructor-arg>
</bean>

Including in the adapter as below: 包括在适配器中如下:

<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
        channel="inboundFileChannel"
        session-factory="sftpSessionFactory"
        local-directory="file:${inbound.local.directory}"
        remote-directory="${inbound.remote.directory}"
        auto-create-local-directory="true"
        delete-remote-files="false"  
        filter="compositeFilter">
    <int:poller max-messages-per-poll="-1" fixed-rate="600000" />
</int-sftp:inbound-channel-adapter>

Update - Changes after adding persistent filter: 更新-添加持久过滤器后的更改:

<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
        channel="inboundFileChannel"
        session-factory="sftpSessionFactory"
        local-directory="file:${inbound.local.directory}"
        remote-directory="file:${inbound.remote.directory}"
        auto-create-local-directory="false"
        delete-remote-files="false"  
        filter="compositeFilter"
        >
    <int:poller max-messages-per-poll="-1" fixed-rate="60000" />
</int-sftp:inbound-channel-adapter>

<bean id="compositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
    <constructor-arg>
        <list>
            <bean class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
                <constructor-arg name="store" ref="metadataStore"/>
                <constructor-arg value="*.*"/>
            </bean>
            <bean class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
                <constructor-arg name="store" ref="metadataStore"/>
                <constructor-arg value="*.*"/>
            </bean>
        </list>
    </constructor-arg>
</bean>

<bean id="redisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
    <property name="port" value="7379"/>
</bean>

<bean name="metadataStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore">
    <constructor-arg name="connectionFactory" ref="redisConnectionFactory"/>
</bean>

This gives me the exception as below. 这给了我如下例外。 I tried the suggestions in the Spring forum, but I don't have a connection issues. 我在Spring论坛中尝试了建议 ,但没有连接问题。

2014-07-17 22:27:14,139 [task-scheduler-1] INFO com.jcraft.jsch - Authentication succeeded (keyboard-interactive). 2014-07-17 22:27:14,139 [task-scheduler-1] INFO com.jcraft.jsch-身份验证成功(键盘交互)。 2014-07-17 22:27:14,192 [task-scheduler-1] INFO com.jcraft.jsch - Disconnecting from localhost port 22 2014-07-17 22:27:14,195 [task-scheduler-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Returning cached instance of singleton bean 'errorChannel' 2014-07-17 22:27:14,197 [task-scheduler-1] DEBUG org.springframework.integration.channel.PublishSubscribeChannel - preSend on channel 'errorChannel', message: [Payload MessagingException content=org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory][Headers={id=84cccecf-20ec-f67b-3f74-a938bf9abf3d, timestamp=1405661234197}] 2014-07-17 22:27:14,197 [task-scheduler-1] DEBUG org.springframework.integration.handler.LoggingHandler - (inner bean)#22 received message: [Payload MessagingException content=org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory][Headers={id=84cccecf-20ec-f67b-3f74-a938bf9 2014-07-17 22:27:14,192 [task-scheduler-1] INFO com.jcraft.jsch-从本地主机端口22断开连接2014-07-17 22:27:14,195 [task-scheduler-1]调试org.springframework .beans.factory.support.DefaultListableBeanFactory-返回单例bean'ErrorChannel'的缓存实例2014-07-17 22:27:14,197 [task-scheduler-1] DEBUG org.springframework.integration.channel.PublishSubscribeChannel-在通道'上预先发送errorChannel”,消息:[有效负载MessagingException内容= org.springframework.messaging.MessagingException:将远程同步到本地目录时发生问题] [Headers = {id = 84cccecf-20ec-f67b-3f74-a938bf9abf3d,timestamp = 1405661234197}] 2014- 07-17 22:27:14,197 [task-scheduler-1]调试org.springframework.integration.handler.LoggingHandler-(内部bean)#22收到消息:[Payload MessagingException content = org.springframework.messaging.MessagingException:发生问题同时将远程同步到本地目录] [Headers = {id = 84cccecf-20ec-f67b-3f74-a938bf9 abf3d, timestamp=1405661234197}] 2014-07-17 22:27:14,199 [task-scheduler-1] ERROR org.springframework.integration.handler.LoggingHandler - org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:193) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(AbstractInboundFileSynchronizingMessageSource.java:167) at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:124) at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:187) at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:52) at org.springframework.integration.endpoint.AbstractPollingEndpoint$1. abf3d,timestamp = 1405661234197}] 2014-07-17 22:27:14,199 [task-scheduler-1]错误org.springframework.integration.handler.LoggingHandler-org.springframework.messaging.MessagingException:将远程同步到本地时发生问题org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(167)上的目录org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:193)上的目录org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:187)处的springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:124)at org.springframework.integration.endpoint.AbstractPollingEndpoint.access $ 000 .java:52),网址为org.springframework.integration.endpoint.AbstractPollingEndpoint $ 1。 call(AbstractPollingEndpoint.java:146) at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143) at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:278) at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49) at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:272) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.Future 在org.springframework.integration.endpoint.AbstractPollingEndpoint $ 1.call(AbstractPollingEndpoint.java:146)在org.springframework.integration.endpoint.AbstractPollingEndpoint $ Poller $ 1.run(AbstractPollingEndpoint.java:278)上致电(AbstractPollingEndpoint.java:143)在org.springframework.integration.util.ErrorHandlingTaskExecutor $ 1.run(ErrorHandlingTaskExecutor.java:52)在org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)在org.springframework.integration.util.ErrorHandlingTaskExecutor处。在org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DeorgatingErrorHandlingRunnable.run(DeorgatingErrorHandlingRunnable.java:54)处org.springframework.integration.endpoint.AbstractPollingEndpoint $ Poller.run(AbstractPollingEndpoint.java:272)处执行(ErrorHandlingTaskExecutor.java:49) .springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)在java.util.concurrent.Executors $ RunnableAdapter.call(Executors.java:439)在java.util.concurrent.Future Task$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) Caused by: org.springframework.messaging.MessagingException: Failed to execute on session at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:311) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:167) ... 21 more Caused by: java.lang.ClassCastException: com.jcraft.jsch.ChannelSftp$LsEntry cannot be c Task $ Sync.innerRun(FutureTask.java:303),位于java.util.concurrent.FutureTask.run(FutureTask.java:138),位于java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.access $ 301(ScheduledThreadPoolExecutor.java:98) java.util.concurrent.ThreadPoolExecutor $ Worker.runTask(ThreadPoolExecutor.java:895)上的java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)在java.util.concurrent.ThreadPoolExecutor $ Worker.run( java.lang.Thread.run(Thread.java:695)上的ThreadPoolExecutor.java:918)原因:org.springframework.messaging.MessagingException:无法在org.springframework.integration.file.remote.RemoteFileTemplate上的会话上执行。在org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:167)处执行(RemoteFileTemplate.java:311)... 21更多原因:java.lang.ClassCastException:com.jcraft.jsch .ChannelSftp $ LsEntry不能为c ast to java.io.File at org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter.fileName(FileSystemPersistentAcceptOnceFileListFilter.java:28) at org.springframework.integration.file.filters.AbstractPersistentAcceptOnceFileListFilter.buildKey(AbstractPersistentAcceptOnceFileListFilter.java:88) at org.springframework.integration.file.filters.AbstractPersistentAcceptOnceFileListFilter.accept(AbstractPersistentAcceptOnceFileListFilter.java:48) at org.springframework.integration.file.filters.AbstractFileListFilter.filterFiles(AbstractFileListFilter.java:40) at org.springframework.integration.file.filters.CompositeFileListFilter.filterFiles(CompositeFileListFilter.java:97) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.filterFiles(AbstractInboundFileSynchronizer.java:157) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:1 与org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter.fileName(FileSystemPersistentAcceptOnceFileListFilter.java:28)上的java.io.File的关系org.springframework.integration.file.org.springframework.integration.file.filters.AbstractPersistentAcceptOnceFileListFilter.accept(AbstractPersistentAcceptOnceFileListFilter.java:48)在org.springframework.integration.file.filters.AbstractFileListFilter.filterFiles(AbstractFileListFilter.java:40)在org.springframework.integration.file。 org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.filterFiles(AbstractInboundFileSynchronizer.java:157)处的filter.CompositeFileListFilter.filterFiles(CompositeFileListFilter.java:97)在org.springframework.integration.file.remote.synchronizer。 .doInSession(AbstractInboundFileSynchronizer.java:1 73) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:167) at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:302) ... 22 more 73)在org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:302)在org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer $ 1.doInSession(AbstractInboundFileSynchronizer.java:167)在org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:302)...另外22个

2014-07-17 22:27:14,199 [task-scheduler-1] DEBUG org.springframework.integration.channel.PublishSubscribeChannel - postSend (sent=true) on channel 'errorChannel', message: [Payload MessagingException content=org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory][Headers={id=84cccecf-20ec-f67b-3f74-a938bf9abf3d, timestamp=1405661234197}] 2014-07-17 22:27:14,199 [task-scheduler-1]调试org.springframework.integration.channel.PublishSubscribeChannel-通道'errorChannel'上的postSend(sent = true),消息:[有效负载MessagingException content = org.springframework .messaging.MessagingException:将远程同步到本地目录时发生问题[Headers = {id = 84cccecf-20ec-f67b-3f74-a938bf9abf3d,timestamp = 1405661234197}]

Update post removing FileSystemPersistentAcceptOnceFileListFilter filter leaving the 'SftpPersistentAcceptOnceFileListFilter' 更新后删除FileSystemPersistentAcceptOnceFileListFilter过滤器,并保留“ SftpPersistentAcceptOnceFileListFilter”

2014-07-18 09:29:21,942 [task-scheduler-3] INFO com.jcraft.jsch - Authentications that can continue: publickey,keyboard-interactive,password 2014-07-18 09:29:21,942 [task-scheduler-3] INFO com.jcraft.jsch - Next authentication method: publickey 2014-07-18 09:29:21,942 [task-scheduler-3] INFO com.jcraft.jsch - Authentications that can continue: keyboard-interactive,password 2014-07-18 09:29:21,942 [task-scheduler-3] INFO com.jcraft.jsch - Next authentication method: keyboard-interactive 2014-07-18 09:29:22,022 [task-scheduler-3] INFO com.jcraft.jsch - Authentication succeeded (keyboard-interactive). 2014-07-18 09:29:21,942 [task-scheduler-3] INFO com.jcraft.jsch-可以继续进行的身份验证:公钥,键盘交互,密码2014-07-18 09:29:21,942 [task-scheduler -3] INFO com.jcraft.jsch-下一认证方法:publickey 2014-07-18 09:29:21,942 [task-scheduler-3] INFO com.jcraft.jsch-可以继续的认证:键盘交互,密码2014 -07-18 09:29:21,942 [task-scheduler-3] INFO com.jcraft.jsch-下一步身份验证方法:交互式键盘2014-07-18 09:29:22,022 [task-scheduler-3] INFO com。 jcraft.jsch-身份验证成功(键盘交互)。 2014-07-18 09:29:22,067 [task-scheduler-3] DEBUG org.springframework.data.redis.core.RedisConnectionUtils - Opening RedisConnection 2014-07-18 09:29:22,068 [task-scheduler-3] INFO com.jcraft.jsch - Disconnecting from localhost port 22 2014-07-18 09:29:22,068 [task-scheduler-3] DEBUG org.springframework.integration.channel.PublishSubscribeChannel - preSend on channel 'errorChannel', message: [Payload MessagingException content=org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory][Headers={id=55304e38-6f82-8350-e763-0f5bcb507788, timestamp=1405700962068}] 2014-07-18 09:29:22,068 [Connect thread localhost session] INFO com.jcraft.jsch - Caught an exception, leaving main loop due to Socket closed 2014-07-18 09:29:22,068 [task-scheduler-3] DEBUG org.springframework.integration.handler.LoggingHandler - (inner bean)#22 received message: [Payload MessagingException content=org.springframework.messaging.MessagingException: Problem occu 2014-07-18 09:29:22,067 [task-scheduler-3]调试org.springframework.data.redis.core.RedisConnectionUtils-打开RedisConnection 2014-07-18 09:29:22,068 [task-scheduler-3]信息com.jcraft.jsch-与本地主机端口22断开连接2014-07-18 09:29:22,068 [task-scheduler-3] DEBUG org.springframework.integration.channel.PublishSubscribeChannel-在“ errorChannel”频道上预先发送,消息:[有效负载MessagingException content = org.springframework.messaging.MessagingException:将远程同步到本地目录时出现问题] [Headers = {id = 55304e38-6f82-8350-e763-0f5bcb507788,timestamp = 1405700962068}] 2014-07-18 09:29: 22,068 [Connect thread localhost session] INFO com.jcraft.jsch-捕获了一个异常,由于Socket关闭而使主循环退出2014-07-18 09:29:22,068 [task-scheduler-3] DEBUG org.springframework.integration.handler .LoggingHandler-(内部bean)#22收到消息:[有效负载MessagingException content = org.springframework.messaging.MessagingException:问题occu rred while synchronizing remote to local directory][Headers={id=55304e38-6f82-8350-e763-0f5bcb507788, timestamp=1405700962068}] 2014-07-18 09:29:22,069 [task-scheduler-3] ERROR org.springframework.integration.handler.LoggingHandler - org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:193) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(AbstractInboundFileSynchronizingMessageSource.java:167) at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:124) at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:187) at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(Abstract 在同步远程到本地目录时rred [Headers = {id = 55304e38-6f82-8350-e763-0f5bcb507788,timestamp = 1405700962068}] 2014-07-18 09:29:22,069 [task-scheduler-3]错误org.springframework .integration.handler.LoggingHandler-org.springframework.messaging.MessagingException:在orgspring上将远程同步到org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:193)上的本地目录时发生问题。 org.springframeworkPos.integration.endpoint.end.org上的.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(AbstractInboundFileSynchronizingMessageSource.java:167)在org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:124) (AbstractPollingEndpoint.java:187)at org.springframework.integration.endpoint.AbstractPollingEndpoint.access $ 000(摘要 PollingEndpoint.java:52) at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146) at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143) at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:278) at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49) at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:272) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) at java.ut org.springframework.integration.endpoint.AbstractPollingEndpoint $ 1.call(AbstractPollingEndpoint.java:146)上org.springframework.integration.endpoint.AbstractPollingEndpoint $ 1.call(AbstractPollingEndpoint.java:143)上的PollingEndpoint.java:52) org.springframework.integration.util.ErrorHandlingTaskExecutor $ 1.run(ErrorHandlingTaskExecutor.java:52)的org.springframework.core.task.SyncTaskExecutor.execute的.integration.endpoint.AbstractPollingEndpoint $ Poller $ 1.run(AbstractPollingEndpoint.java:278) (SyncTaskExecutor.java:50)在org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)在org.springframework.integration.endpoint.AbstractPollingEndpoint $ Poller.run(AbstractPollingEndpoint.java:272)在org。 org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)上的springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)在java.ut il.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) Caused by: org.springframework.messaging.MessagingException: Failed to execute on session at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:311) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:167) ... 2 il.concurrent.Executors $ RunnableAdapter.call(Executors.java:439)在java.util.concurrent.FutureTask $ Sync.innerRun(FutureTask.java:303)在java.util.concurrent.FutureTask.run(FutureTask.java: 138)在java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)在java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)在java.util.concurrent。 java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:918)处的Worker.runTask(ThreadPoolExecutor.java:895)来自java.lang.Thread.run(Thread.java:695)的原因:org.springframework .messaging.MessagingException:无法在org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(Abstract java:167)... 2 1 more Caused by: org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; 1个以上原因:org.springframework.data.redis.RedisConnectionFailureException:无法获取Jedis连接; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:97) at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143) at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41) at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128) at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91) at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:177) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152) at org.springframework.data.redi 嵌套的异常是redis.clients.jedis.exceptions.JedisConnectionException:无法从org.springframework.data上的org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:97)的池中获取资源。 org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)上的.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)在org.springframework.data.redis.core上org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)在org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)在org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java :78)在org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:177)在org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)在org.springframework。 data.redi s.core.AbstractOperations.execute(AbstractOperations.java:84) at org.springframework.data.redis.core.DefaultHashOperations.putIfAbsent(DefaultHashOperations.java:179) at org.springframework.data.redis.core.DefaultBoundHashOperations.putIfAbsent(DefaultBoundHashOperations.java:91) at org.springframework.data.redis.support.collections.RedisProperties.putIfAbsent(RedisProperties.java:228) at org.springframework.integration.redis.metadata.RedisMetadataStore.putIfAbsent(RedisMetadataStore.java:139) at org.springframework.integration.file.filters.AbstractPersistentAcceptOnceFileListFilter.accept(AbstractPersistentAcceptOnceFileListFilter.java:51) at org.springframework.integration.file.filters.AbstractFileListFilter.filterFiles(AbstractFileListFilter.java:40) at org.springframework.integration.file.filters.CompositeFileListFilter.filterFiles(CompositeFileListFilter.java:97) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.filterFiles(AbstractInbound org.springframework.data.redis.core.DefaultBoundHashOperations.putIfAbsent(s.core.AbstractOperations.execute(AbstractOperations.java:84)在org.springframework.data.redis.core.DefaultHashOperations.putIfAbsent(DefaultHashOperations.java:179)在org.springframework.data.redis.core.DefaultBoundHashOperations.putIfAbsent( org.springframework.data.redis.support.collections.RedisProperties.putIfAbsent(RedisProperties.java:228)的DefaultBoundHashOperations.java:91)org.springframework.integration.redis.metadata.RedisMetadataStore.putIfAbsent(RedisMetadataStore.java:139)在org.springframework.integration.file.filters.AbstractPersistentAcceptOnceFileListFilter.accept(AbstractPersistentAcceptOnceFileListFilter.java:51)在org.springframework.integration.file.filters.AbstractFileListFilter.filterFiles(AbstractFileListFilter.java:40)在org.springframework.integration.file org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.filterFiles(AbstractInbound)上的.filters.CompositeFileListFilter.filterFiles(CompositeFileListFilter.java:97) FileSynchronizer.java:157) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:173) at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:167) at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:302) ... 22 more Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool at redis.clients.util.Pool.getResource(Pool.java:42) at redis.clients.jedis.JedisPool.getResource(JedisPool.java:84) at redis.clients.jedis.JedisPool.getResource(JedisPool.java:10) at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90) ... 41 more Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused at redis.clients.jedis org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer $ 1.doInSession(AbstractInboundFileSynchronizer.java:173)上的org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer(In1.doInIn java:167),位于org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:302)... 22更多原因:redis.clients.jedis.exceptions.JedisConnectionException:无法从在redis.clients.jedis.JedisPool.getResource(JedisPool.java:84)的redis.clients.jedis.JedisPool.getResource(JedisPool.java :)的redis.clients.util.Pool.getResource(Pool.java:42)处创建池10)位于org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90)... 41更多原因:redis.clients.jedis.exceptions.JedisConnectionException:java.net.ConnectException:Connection在redis.clients.jedis拒绝 .Connection.connect(Connection.java:150) at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:71) at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1783) at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:65) at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:819) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:429) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360) at redis.clients.util.Pool.getResource(Pool.java:40) ... 44 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:382) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:241) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:228) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:431) at java.net.Socket.connect(Socket.java:527) at red 在redis.clients.jedis上的.Connection.connect(Connection.java:150)在redis.clients.jedis上的redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1783)上的。 org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.org)上的jedis.JedisFactory.makeObject(JedisFactory.java:65)在org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:819)在org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool。 java:429),位于org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360),位于redis.clients.util.Pool.getResource(Pool.java:40)... ... 44多由以下原因引起: java.net.ConnectException:在java.net.PlainSocketImpl.docket(PlainSocketImpl.java:382:java.net.PlainSocketImpl.docketConnect(Native Method)处的连接被拒绝。java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:241)在java.net.PlainSocketImpl.connect(PlainSocketImpl.java:228)在java.net.SocksSocketImpl.connect(SocksSocketImpl.java:431)在java.net.Socket.connect(Socket.java:527)在红色 is.clients.jedis.Connection.connect(Connection.java:144) ... 51 more is.clients.jedis.Connection.connect(Connection.java:144)...还有51个

2014-07-18 09:29:22,069 [task-scheduler-3] DEBUG org.springframework.integration.channel.PublishSubscribeChannel - postSend (sent=true) on channel 'errorChannel', message: [Payload MessagingException content=org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory][Headers={id=55304e38-6f82-8350-e763-0f5bcb507788, timestamp=1405700962068}] 2014-07-18 09:29:22,069 [task-scheduler-3]调试org.springframework.integration.channel.PublishSubscribeChannel-通道'errorChannel'上的postSend(sent = true),消息:[有效负载MessagingException content = org.springframework .messaging.MessagingException:将远程同步到本地目录时发生问题[Headers = {id = 55304e38-6f82-8350-e763-0f5bcb507788,timestamp = 1405700962068}

It could be the case that if the new files dropped on the server has same name(which are downloaded in the first iteration) then spring integration will not attempt to download the files again. 可能的情况是,如果放置在服务器上的新文件具有相同的名称(在第一次迭代中下载),则spring集成将不会尝试再次下载文件。 "acceptOnceFileListFilter" seems like something that would restrict sftp integration component to download the same file again. “ acceptOnceFileListFilter”似乎会限制sftp集成组件再次下载同一文件。

Update : Here is some sample code which polls a remote directory every 10 secs without introducing any filters(I am not sure why you need them, as you want to download the files regardless). 更新 :以下是一些示例代码,该示例代码每10秒轮询一次远程目录,而没有引入任何过滤器(我不确定为什么要使用它们,因为无论如何都要下载文件)。 Given that you dont want to change the file names, the only thing that you will have to make sure is that your downloaded local needs to deleted when the next file is dropped on the remote server. 鉴于您不想更改文件名,因此唯一需要确保的是,当下一个文件放在远程服务器上时,需要删除下载的本地文件。 I verified it and worked fine for me. 我验证了它,并为我很好地工作。

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns:int="http://www.springframework.org/schema/integration"
   xmlns:int-ftp="http://www.springframework.org/schema/integration/ftp"
   xsi:schemaLocation="http://www.springframework.org/schema/beans      http://www.springframework.org/schema/beans/spring-beans.xsd
   http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd

   http://www.springframework.org/schema/integration/ftp http://www.springframework.org/schema/integration/ftp/spring-integration-ftp.xsd">


<int:channel id="incomingChannel"/>

<bean id="ftpClientFactory" class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
    <property name="host" value="host"/>
    <property name="username" value="username"/>
    <property name="password" value="password"/>
</bean>

<!-- poll every 10 secs without worrying about file names-->

<int-ftp:inbound-channel-adapter id="triggerFtpInBound"
                                 channel="incomingChannel"
                                 auto-create-local-directory="true"
                                 local-directory="C:\home\temp\ftp"
                                 remote-directory="/export/home/rbaljinder/ftp-test"
                                 filename-pattern="*.*"
                                 session-factory="ftpClientFactory">
    <int:poller cron="1/10 * * * * *" max-messages-per-poll="1"/>
</int-ftp:inbound-channel-adapter>

UPDATE: seems working now.Added local-filter="acceptAllFileListFilter". 更新:似乎现在正在工作。添加了local-filter =“ acceptAllFileListFilter”。

 <int-ftp:inbound-channel-adapter id="triggerFtpInBound"
                                 channel="incomingChannel"
                                 auto-create-local-directory="true"
                                 local-directory="C:\home\temp\ftp"
                                 remote-directory="/export/home/cwk2/ftp-test"
                                 filename-pattern="*.*"
                                 session-factory="ftpClientFactory"
                                 local-filter="acceptAllFileListFilter">
    <int:poller cron="1/10 * * * * *" max-messages-per-poll="1"/>
</int-ftp:inbound-channel-adapter>
<bean id="acceptAllFileListFilter"
      class="org.springframework.integration.file.filters.AcceptAllFileListFilter"/>
<int:service-activator id="jobServiceActivator"
                       input-channel="incomingChannel"
                       ref="triggerJobLauncher"
                       method="launch"/>

@Component("triggerJobLauncher")
public static class TriggerJobLauncher {

    @Autowired
    JobLauncher jobLauncher;

    public void launch(File file) throws Exception {
        System.out.println("test:" + file);
    }

}

Gist Here 要点在这里

Hopefully self-explanatory; 希望能自我解释; it just prints the file name on the console, then deletes it. 它只是在控制台上打印文件名,然后将其删除。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM