[英]Constructing RequestBodyStream from Lazy ByteString when length is known
I am trying to adapt this AWS S3 upload code to handle Lazy ByteString
where length is already known (so that it is not forced to be read in its entirety in memory - it comes over the network where length is sent beforehand). 我正在尝试修改此AWS S3上传代码以处理长度已知的Lazy ByteString
(这样就不会强制在内存中完整读取它-它是通过网络预先发送长度的)。 It seems I have to define a GivesPopper
function over Lazy ByteString
to convert it to RequestBodyStream
. 看来我必须在Lazy ByteString
上定义GivesPopper
函数,以将其转换为RequestBodyStream
。 Because of the convoluted way GivesPopper
is defined, I am not sure how to write it for Lazy ByteString
. 由于定义了GivesPopper
的复杂方式,因此我不确定如何为Lazy ByteString
编写它。 Will appreciate pointers on how to write it. 将不胜感激如何编写它的指针。 Here is how it is written for reading from the file: 这里是它是如何从文件中读取写入:
let file ="test"
-- streams large file content, without buffering more than 10k in memory
let streamer sink = withFile file ReadMode $ \h -> sink $ S.hGet h 10240
streamer
in the code above is of type GivesPopper ()
if I understand it correctly. 如果我正确理解,上面代码中的streamer
GivesPopper ()
类型为GivesPopper ()
。 Given a Lazy ByteString
with known length len
, what would be a good way to write GivesPopper
function over it? 给定一个长度为len
的Lazy ByteString
,在上面编写GivesPopper
函数的一种好方法是什么? We can read one chunk at a time. 我们一次可以读取一个块。
Is this what you're looking for? 这是您要找的东西吗?
import qualified Data.ByteString as S
import qualified Data.ByteString.Lazy as L
import System.IO
file = "test"
-- original streamer for feeding a sink from a file
streamer :: (IO S.ByteString -> IO r) -> IO r
streamer sink = withFile file ReadMode $ \h -> sink $ S.hGet h 10240
-- feed a lazy ByteString to sink
lstreamer :: L.ByteString -> (IO S.ByteString -> IO r) -> IO r
lstreamer lbs sink = sink (return (L.toStrict lbs))
lstreamer
type checks but probably doesn't do exactly what you want it to do. lstreamer
类型检查,但可能并没有完全按照您想要的做。 It simply returns the same data every time the sink calls it. 每次接收器调用时,它仅返回相同的数据。 On the other hand S.hGet h ...
will eventually return the empty string. 另一方面, S.hGet h ...
最终将返回空字符串。
Here is a solution which uses an IORef to keep track of if we should start returning the empty string: 这是一个使用IORef来跟踪是否应该开始返回空字符串的解决方案:
import Data.IORef
mklstream :: L.ByteString -> (IO S.ByteString -> IO r) -> IO r
mklstream lbs sink = do
ref <- newIORef False
let fetch :: IO S.ByteString
fetch = do sent <- readIORef ref
writeIORef ref True
if sent
then return S.empty
else return (L.toStrict lbs)
sink fetch
Here fetch
is the action which gets the next chunk. 这里fetch
是获取一个块的动作。 The first time you call it you will get the original lazy Bytestring (strict-ified). 第一次调用它,您将获得原始的惰性Bytestring(严格限制)。 Subsequent calls will always return the empty string. 后续调用将始终返回空字符串。
Update 更新
Here's how to give out a small amount at a time: 一次发放少量款项的方法如下:
mklstream :: L.ByteString -> (IO S.ByteString -> IO r) -> IO r
mklstream lbs sink = do
ref <- newIORef (L.toChunks lbs)
let fetch :: IO S.ByteString
fetch = do chunks <- readIORef ref
case chunks of
[] -> return S.empty
(c:cs) -> do writeIORef ref cs
return c
sink fetch
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.