[英]Upload a InputStream to AWS s3 asynchronously (non-blocking) using AWS SDK for Java, version 2
When I am uploading inputStream
object to s3 synchronously (blocking way) it works.当我将inputStream
object 同步(阻塞方式)上传到 s3 时,它可以工作。
S3Client s3Client = S3Client.builder().build();
s3Client.putObject(objectRequest, RequestBody.fromInputStream(inputStream,STREAM_SIZE));
but when I try the same with S3AsyncClient
there is no .fromInputStream
method on AsyncRequestBody
.但是当我对AsyncRequestBody
进行相同尝试时, S3AsyncClient
上没有.fromInputStream
方法。
S3AsyncClient s3AsyncClient = S3AsyncClient.builder().build();
s3AsyncClient.putObject(objectRequest, AsyncRequestBody.fromInputStream(inputStream,STREAM_SIZE)); // error no method named 'fromInputStream'
And I can't use .fromByteBuffer
as it will load the entire stream into memory, which I don't want.而且我不能使用.fromByteBuffer
,因为它会将整个 stream 加载到 memory 中,这是我不想要的。
I am interested why there is no method to read from InputStream in AsyncRequestBody
.我很感兴趣,为什么没有从AsyncRequestBody
中的 InputStream 读取的方法。 And Is there any alternatives?还有其他选择吗?
For anyone using Kotlin and coroutines: here is a kotlin wrapper which will create an asynchronous AsyncRequestBody
from an InputStream
.对于使用 Kotlin 和协程的任何人:这是一个 kotlin 包装器,它将从InputStream
创建一个异步AsyncRequestBody
。 The wrapper will run in a background thread by default, but you can pass an explicit CoroutineScope
and run it inside of your coroutine, which will avoid creating a separate thread.默认情况下,包装器将在后台线程中运行,但您可以传递显式CoroutineScope
并在协程内部运行它,这将避免创建单独的线程。
import io.ktor.util.cio.*
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.DelicateCoroutinesApi
import kotlinx.coroutines.GlobalScope
import kotlinx.coroutines.launch
import org.reactivestreams.Subscriber
import org.reactivestreams.Subscription
import software.amazon.awssdk.core.async.AsyncRequestBody
import java.io.InputStream
import java.nio.ByteBuffer
import java.util.*
@OptIn(DelicateCoroutinesApi::class)
class StreamAsyncRequestBody(
inputStream: InputStream,
private val coroutineScope: CoroutineScope = GlobalScope
) :
AsyncRequestBody {
private val inputChannel =
inputStream.toByteReadChannel(context = coroutineScope.coroutineContext)
override fun subscribe(subscriber: Subscriber<in ByteBuffer>) {
subscriber.onSubscribe(object : Subscription {
private var done: Boolean = false
override fun request(n: Long) {
if (!done) {
if (inputChannel.isClosedForRead) {
complete()
} else {
coroutineScope.launch {
inputChannel.read {
subscriber.onNext(it)
if (inputChannel.isClosedForRead) {
complete()
}
}
}
}
}
}
private fun complete() {
subscriber.onComplete()
synchronized(this) {
done = true
}
}
override fun cancel() {
synchronized(this) {
done = true
}
}
})
}
override fun contentLength(): Optional<Long> = Optional.empty()
}
Example usage:示例用法:
suspend fun s3Put(objectRequest: PutObjectRequest, inputStream: InputStream) = coroutineContext {
s3Client.putObject(objectRequest, StreamAsyncRequestBody(inputStream, this)
}
If you use Java, you will need to create your own wrapper and use a different coroutine library.如果您使用 Java,您将需要创建自己的包装器并使用不同的协程库。 Alternatively, you could create an Executor
with a fixed number of threads: if you have too many uploads running at once, they will block one another, but they won't create too many threads and block the entire program.或者,您可以创建一个具有固定数量线程的Executor
:如果您一次运行的上传太多,它们会相互阻塞,但它们不会创建太多线程并阻塞整个程序。
EDIT: Fixed the code.编辑:修复了代码。 I didn't test the previous version, I tested this version a few times to upload and it worked.我没有测试之前的版本,我测试了这个版本几次上传,它可以工作。 Of course that doesn't mean it's bug-free though:)当然,这并不意味着它没有错误:)
After some research this is what I found:经过一番研究,这是我发现的:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.