IO Operations in Conduit?

Question

This is probably a pretty obscure case, but please bear with me.

I'm writing a web crawler that needs to download webpages and save them to local disk. Meanwhile, being a crawler, it needs to find the hyperlinks on each page and download those linked pages as well.

Right now I'm looking at the first sample code given on this page . I'm trying to write a conduit of this type:

processLink :: Conduit S.ByteString IO S.ByteString

And then I will replace this line in the sample code:

responseBody res $$+- sinkFile "foo.txt"

with this line:

responseBody res $= processLinks $$+ sinkFile

So what I'm trying to do is to implement a conduit called processLink, which processes a ByteString, extracks links from them, and then pass the original ByteString along to sinkFile so that the page can be saved. Then I can call functions on those links recursively to download those linked pages.

However, since processLink is a pure function, it seems like I can't do any IO operations inside processLink, and therefore cannot do the things I need like downloading those linked pages. Or can I? What could I do to get around this problem?

Answer 1

Alright I figured it out.... I think what I need is just liftIO lol. Sorry I'm kinda new to Haskell :P