Writing into a file in a streaming way and stop when we reach a given memory file size

Question

Let's say I want to save a stream of MyItem into a file (in JSON for example). I would like to stop when the file reach a certain size limit in bytes. I would like to do it in Haskell... streaming is not the issue for me it's more how to get the file size information after each item put into this file...

Answer 1

I don't think pipes anything for this out of the box, but it's easy to write your own toHandleBounded .

On each step you check the file size and if it's less than the limit you loop, writing another block to the file.

toHandleBounded :: MonadIO m => Integer -> Handle -> Consumer' ByteString m ()
toHandleBounded limit hd = do
  current <- liftIO $ hTell hd
  when (current < limit) $ do
    x <- await
    liftIO $ hPut hd x
    toHandleBounded limit hd

main :: IO ()
main =
  withFile "/dev/urandom" ReadMode $ \hIn ->
  withFile "out.txt" WriteMode $ \hOut ->
    runEffect $ P.fromHandle hIn >-> toHandleBounded (1024 * 500 ) hOut