Unable to click on <a> tag using Puppeteer

Question

I am trying to scrape a real estate website. This is my first time using Puppeteer. Up to now I have managed to open the link and click on the "accept all" button for cookies. Now I am trying to click on the "sign in" button. This is my code:

const puppeteer = require("puppeteer");

const link1 =
  "https://www.daft.ie/for-rent/walled-garden-wyckham-way-dundrum-dublin-14/3988400";

(async () => {
  const browser = await puppeteer.launch({
    headless: false,
    defaultViewport: null,
  });
  const page = await browser.newPage();
  await page.goto(link1);

  const elements = await page.$x("//button[normalize-space()='Accept All']");
  await elements[0].click();
  console.log("cookies accepted");

  const signin = await page.$x("//a[contains(., 'Sign in')]");
  await signin[0].click();

  console.log("Sign in clicked");

  await browser.close();
})();

This is the error I get after I run the script:

 throw new Error('Execution context was destroyed, most likely because of a navigation.');
              ^

Error: Execution context was destroyed, most likely because of a navigation.
    at rewriteError (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:280:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async ExecutionContext._ExecutionContext_evaluate (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:226:56)
    at async ElementHandle.evaluateHandle (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/JSHandle.js:97:16)
    at async Object.internalHandler.queryAll (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/QueryHandler.js:63:30)
    at async ElementHandle.$$ (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/ElementHandle.js:116:17)
    at async /Users/alex/Desktop/coding/daft/index.js:18:1

I verified the xPath I am using for the a tag sign in button is correct.

Here's the HTML of the sign in link:

<a data-testid="top-level-active-nav-link" href="/auth/authenticate" class="Navigationstyled__TopLevelItemActiveLink-sc-1tagpb7-6 bsFoce">Sign in</a>

What am I doing wrong here?

Link of the target page - list

Answer 1

page.click() issues a trusted event that sometimes fails due to visibility issues. A workaround is using element.evaluate(el => el.click()) to trigger a native DOM click.

Secondly, you'll want to waitForNavigation using Promise.all when triggering an event that initiates a navigation.

Putting it together:

const puppeteer = require("puppeteer"); // ^16.2.0

let browser;
(async () => {
  browser = await puppeteer.launch({headless: false});
  const [page] = await browser.pages();
  const url =
    "https://www.daft.ie/for-rent/walled-garden-wyckham-way-dundrum-dublin-14/3988400";
  await page.goto(url, {waitUntil: "domcontentloaded"});
  const xp = "//button[normalize-space()='Accept All']";
  const cookiesBtn = await page.waitForXPath(xp);
  await cookiesBtn.click();
  const signIn = await page.waitForXPath("//a[contains(., 'Sign in')]");
  await Promise.all([
    signIn.evaluate(el => el.click()),
    page.waitForNavigation()
  ]);
  await page.waitForSelector("#username");
  await page.type("#username", "hello");
  await page.type("#password", "world");
  await page.click('input[name="login"]');
  await page.waitForTimeout(100000); // pause just to show it's working
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

Note that the sign-in link is https://www.daft.ie/auth/authenticate , so you may be able to achieve your goals more reliably and efficiently if you use this as your root URL.

Another trick I use for scripts with complex sign-in workflows that involve a lot of frames, oauth and other difficult-to-automate components is to do it all by hand in Puppeteer's browser while saving the cookies and site data in the userDataDir . It seems that this website should be workable without that trick but it's worth mentioning in case you run into trouble.