Web crawling in F#

Question

I'm working on web crawler in F# and have some question on the subject.

Do you need to create a script to crawl in F# or can you create a console application? I have seen some tutorials, and all of them uses #r (references), which is not possible in a console application? What are the benefits with using F# for web crawling maybe compared to C#? I hope someone can shed some light on these questions.

Answer 1

I wrote a web crawler a few years ago. I used F#. I wrote as a console application. the #r referenced are used in F# script which is cool but I typically deploy a binary.

I picked F# over C# because it's more productive to me (that is subjective of course) but also because I was developed a DSL in F# to describe what parts of a page interested me.

I used FSharp.Data to do HTML parsing: http://fsharp.github.io/FSharp.Data/library/HtmlParser.html