Spring Cloud DataFlow 在 AWS S3 存储桶源中的新文件后使用启动任务

Question

I'm trying to create a batch processing started by a new file in AWS S3.

So the flow is:

1 - A new file is uploaded to AWS S3 Bucket
2 - SCDF detects
3 - SCDF launch the task (Spring Batch application)
4 - Spring Batch application process the file and stores in a DB.

Something similar to this, but with S3 Bucket: https://dataflow.spring.io/docs/recipes/batch/sftp-to-jdbc/

Maybe is a misunderstood of the concept, but in SFTP Source I could set Port, Host, User and Pass, but in S3 Source I doesn't have region and credentials properties.

Where do I set the AWS properties?

Answer 1

There's an Amazon AWS common options section in the README (see: old-app / new-app ), which includes the common AWS-specific properties one can override.

You can pass them as inline properties in the stream definition or when deploying the stream by following the deployer properties convention.

Answer 2

I provided a detailed example of this use case here (with Minio S3). This works out of the box with the latest release of stream applications, but will require some customization if you are using previous versions.