how to remove the filtered
lines from the current streaming file using fs2 and get the count of filtered lines as the return type?
ex: If the old.txt
contains strings separated by a newline (\\n):
john
sam
chen
yval
....
and val myList = List("chen","yval")
.
def converter[F[_]](implicit F: Sync[F]): F[Unit] =
io.file.readAll[F](Paths.get("testdata/old.txt"), 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s => myList.contains(s))//remove this from the old file and write to new file
.intersperse("\n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/new.txt")))
.compile.drain
// at the end of the universe...
val u: Unit = converter[IO].unsafeRunSync()
You can use the observe method of the Stream
class.
You are looking for a function def converter[F[_]: Sync]: F[Int]
, which produces a computation F[Int]
whose result (of type Int
) is the number of filtered lines, and whose effect is to write those lines to the output file. To follow a plumbing analogy, you want to feed your filtered stream to two outputs, one for the result, and one for the effect. You can do this using the function observe , defined as
def observe(sink: Sink[F, O])(implicit F: Effect[F], ec: ExecutionContext): Stream[F, O]
A Sink[F,O]
is an alias for a function Stream[F, O] => Stream[F, Unit]
. In your case, the sink is the part of your code that writes your filtered stream into the output file:
def writeToFile[F[_]: Sync]: Sink[F, String] =
_.intersperse("\n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/new.txt")))
The other output is to reduce, or rather fold,
def converter[F[_]](implicit F: Effect[F], ec: ExecutionContext): F[Int] =
io.file.readAll[F](Paths.get("testdata/old.txt"), 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s => myList.contains(s))
.observe(writeToFile[F])
.compile
.fold[Int](0)( (c, _) => c+1)
}
Note : for this solution you need to restrict the type-class of F
to be Effect
, and you need to use an ExecutionContext
. The fold
is defined in the ToEffect
class.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.