Scrape web site generated by Javascript

Question

I think this is a real challenging one!

I write a website for my local football league, www.rdyfl.co.uk , and include javascript code snippets from the FA's Full-Time system where we generate our fixtures, linking in tables fixtures recent results etc.

For another feature I want to add to the site I need to scrape the 'Upcoming Fixtures' for each agegroup and division but when I examine the source I have two problems.

1. The fixtures content is generated by javascript and therefore I need to see the generated source and not just the source.

2. When I view the generated source using Firefox the team names are actually further javascript links and not the name itself.

I basically want to somehow download the fixtures on a regular basis and write then to a mysql database ?

I have asked the FA and they have no more options available to access the data ?

Having never coded for scraping before can anyone point me to a simple solution or does anyone fancy the challange?

Answer 1

This question was asked a long time ago, but I noticed it was active today 🤷.

You should be able to scrape the website using a headless browser such as Puppeteer . Using Puppeteer you are able to access a URL and execute JavaScript or interact with the website as you would with an ordinary browser. Parsing the output DOM and storing it should then be relatively straightforward.

There are plenty of articles on this topic using Puppeteer.

Answer 2

The latest version of OutWit Hub is doing a pretty good job on dynamic content. The source scraped by outwit to extract links, images, documents and tables and text is the updated DOM. You can certainly make a job to grab what you need using these. Custom scrapers are still applied to the static source in version 1.0.3 but version 1.1.x (still in beta) will offers the choice between the static source and the dynamically modified DOM.

Answer 3

Scrapping content produced by Javascript is challenging. AFAIK you will need to do this with AJAX. Hopefully the content has some css that you can grab with jQuery or at least some id's. Do you have id's or classes that you can grab?