Stream URL to a local file with fs2

Question

Using fs2 (ver. 1.0.4) and cats-effect IO , I can stream an URL to a local file,

import concurrent.ExecutionContext.Implicits.global

def download(spec: String, filename: String): Stream[IO, Unit] = 
  io.readInputStream((new URL(spec).openConnection.getInputStream), 4096, global, true)
    .through(io.file.writeAll(Paths.get(filename), global))

However, this code snippet does not return any information about the process when it is completed. On top of that, besides knowing the operation is successful or failure, I also want to know how many bytes are read if the operation is a success. I do not want to check the new file size to get this information. On the other hand, if the operation is a failure, I want to know what causes the failure.

I tried attempt but I could not resolve the subsequent steps to write the raw bytes to the new file. Please advise. Thanks

Answer 1

You might want to play around with observe . I'm sure there's a better way to do this, but here's an example that might help you get unstuck:

Your original code that compiles and runs:

import fs2.io
import cats.effect.{IO, ContextShift}
import concurrent.ExecutionContext.Implicits.global

import java.net.URL
import java.nio.file.Paths

object Example1 {
  implicit val contextShift: ContextShift[IO] = IO.contextShift(global)

  def download(spec: String, filename: String): fs2.Stream[IO, Unit] =
    io.readInputStream[IO](IO(new URL(spec).openConnection.getInputStream), 4096, global, closeAfterUse=true)
      .through(io.file.writeAll(Paths.get(filename), global))

  def main(args: Array[String]): Unit = {
    download("https://isitchristmas.com/", "/tmp/christmas.txt")
      .compile.drain.unsafeRunSync()
  }
}

Using observe to count bytes:

import fs2.io
import cats.effect.{IO, ContextShift}
import concurrent.ExecutionContext.Implicits.global

import java.net.URL
import java.nio.file.Paths

object Example2 {
  implicit val contextShift: ContextShift[IO] = IO.contextShift(global)

  final case class DlResults(bytes: Long)

  def download(spec: String, filename: String): fs2.Stream[IO, DlResults] =
    io.readInputStream[IO](IO(new URL(spec).openConnection.getInputStream), 4096, global, closeAfterUse = true)
      .observe(io.file.writeAll(Paths.get(filename), global))
      .fold(DlResults(0L)) { (r, _) => DlResults(r.bytes + 1) }

  def main(args: Array[String]): Unit = {
    download("https://isitchristmas.com/", "/tmp/christmas.txt")
      .compile
      .fold(()){ (_, r) => println(r)}
      .unsafeRunSync()
  }
}

Output:

> DlResults(42668)

Answer 2

I have found the solution in terms of Resource and IO and, @codenoodle suggestion.

Update #1

Resource is removed because it is redundant when used with FS2 and complicates the code.

import java.io.{
  FileNotFoundException,
  FileOutputStream,
  InputStream,
  OutputStream
}

import java.net.URL

import cats.effect.{ExitCode, IO, IOApp, Resource}
import cats.implicits._
import fs2._

import scala.concurrent.ExecutionContext.Implicits.global

object LetStream extends IOApp {
  override def run(args: List[String]): IO[ExitCode] = {
    def write(source: IO[InputStream], target: IO[OutputStream]) =
      io.readInputStream(source, 4096, global)
        .chunks
        .flatMap(Stream.chunk)
        .observe(io.writeOutputStream(target, global))
        .chunks
        .fold(0L)((acc, chunk) => acc + chunk.size)

    write(IO(new FileOutputStream("image.jpg")), 
      IO(new URL("http://localhost:8080/images/helloworld.jpg")
            .openConnection
            .getInputStream))
      .use(_.compile.toList)
      .flatMap(size => 
        IO(println(s"Written ${size.head} bytes")) *> IO(ExitCode.Success))      
      .recover {
        case t: FileNotFoundException =>
          IO(println(s"Not found, ${t.getMessage}")) *> IO(ExitCode.Error)
        case err =>
          IO(println(err.getMessage)) *> IO(ExitCode.Error)
      }
  }
}