How do I click on an HTML tab item/filter using puppeteer?

Question

I am creating a web-scraping app for a website which uses tab headers to filter the information displayed in a table. I need to select a particular filter before extracting data from the table, but I'm not having any luck clicking on a tab item, whereas I am able to click on a button.

I am using puppeteer and cheerio in this app and I have successfully navigated to a related page and clicked on a button before extracting data, but the tab header doesn't seem to react the same way, although it also requires a human user to click to select it.

This is a snippet of my code:

const page = await browser.newPage();
await page.goto('https://na.op.gg/summoner/champions/userName=' + 'TheJackal666');

const html = await page.content();
const $ = cheerio.load(html);

//This is the troublesome line
await page.click('#SummonerLayoutContent > div.tabItem.Content.SummonerLayoutContent.summonerLayout-champions > div > div > div.Content.tabItems > div.tabItem.season-13 > div > div.stats-filter > div > div:nth-child(2)');
//The scraping function follows

I expect to, when performing the rest of my scraping function, be given results consistent with the information that is displayed when the "Ranked Solo" tab header is active. Instead, it currently fails to activate that selector, and scrapes the data that is displayed when the default "Total" tab header is active.

Thank you so much for any and all suggestions y'all have <3!

Answer 1

it currently fails to activate that selector

The target site seems to be quite heavy, so give it some time to load and execute scripts:

await page.goto('https://na.op.gg/summoner/champions/userName=TheJackal666', { waitUntil : "domcontentloaded" });

const selector = "#SummonerLayoutContent > div.tabItem.Content.SummonerLayoutContent.summonerLayout-champions > div > div > div.Content.tabItems > div.tabItem.season-13 > div > div.stats-filter > div > div:nth-child(2)";

// Wait fo the tab selector to be present
await page.waitFor(selector);

await page.click(selector);

Also not the you use the username as if it were a variable, not a string:

await page.goto('https://na.op.gg/summoner/champions/userName=' + TheJackal666);

If it is not defined anywhere before this will lead to an error.

Finally, when developing such scripts consider first using headful mode (with visible Chromium browser):

const browser = await puppeteer.launch({ headless: false});

This will give you a better understanding at first of what's going on during scrape.