Navigate to URL and grab href attribute from anchor tag?

Question

I have a list of let's say... 10000 URLs. The markup is structured identically but the attributes for some of the DOM elements are unique to the URL.

Is there a was to use JavaScript to:

• Navigate to a URL?
• Grab a DOM element (in my case, pull the href attr for a specific anchor tag)?

Just looking for a push in the right direction. If something like this exists, I am planning to code a loop to iterate through all 10000 URLs and DoSomething() with them.

Answer 1

I recommend using Node.js: look into a Node module like Cheerio , to scrape your URLs and look at their DOM elements. Should be a relatively simple script.

Answer 2

Success! I was able to get there using the WebClient class and the IHTMLDocument2 interface in a C# console application.

WebClient Class Reference

IHTMLDocument2 Interface Reference