如何从具有 AJAX 请求的网站获取 HTML 页面而不出现 CORS 错误？

Question

I'm trying to do basic webscraping with javascript code using XMLHttpRequest. The code is triggered when I click on a button in an html page that I open with my Firefox browser.

The code I'm using is below. It tries to get the html page from ATP tour website and to print it as first step. Then I will try to parse it to get the information I want (like a rank for example).

let htmlRequest = new XMLHttpRequest();
htmlRequest.open("GET", "https://www.atptour.com/en/players/matteo-berrettini/bk40/overview");
htmlRequest.onreadystatechange = function () {
    if (this.readyState == XMLHttpRequest.DONE && this.status == 200) {
        console.log(this.responseText);
    }
};
htmlRequest.send();

I discovered that running this code on localhost does not work because I'm not on the same domain as the site I'm sending my request (it is the classical CORS issue).

I now understand I need to create a web server but my understanding of the problem stops here.

So my question is: what do I have to implement to make my js script working?

Answer 1

Have you tried a web-scraping library? These are a few:

• Apify
• Puppeteer
• Cheerio