'You Look Like a Robot' Error , while using puppeteer in firebase cloud functions

Question

Im using the puppeteer package, in order to scrap a web page data that is fetched by clicking a button in this page

this are the presetting that I'm using:

const puppeteer = require('puppeteer-extra')

const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())

// Add adblocker plugin to block all ads and trackers (saves bandwidth)
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker')
puppeteer.use(AdblockerPlugin({ blockTrackers: true }))

those setting are made in order that I will not be detected as a robot.

here what I'm doing: (basically, creating a request by clicking a button, then this request return a json with a data that fill up some text info in a label, then I'm reading the data from that label

here's how im clicking the button:

const box = await btn.boundingBox();
const x = box.x + (box.width/2);
const y = box.y + (box.height/2);
console.log(x, y);
page.mouse.move(x,y,{step:1});
page.mouse.click(x,y)
await page.waitForTimeout(4000);

then afterwards ---> I'm getting the data from the data:

const [result] = await page.$x('//*[@id="content"]/div[1]/div[1]/div/div[2]/div');
//   const txt = await result.evaluate.toString
let value = await page.evaluate(el => el.textContent, result);
console.log(value);
console.log('done?');
await browser.close();
const dic = {};
dic['status'] = 200;
dic['data'] = {"message": value};
  
response.send(dic);

I'm also using the 'on' method in order to see if the im getting a response from the action of clicking the button, like so:

await page.on('response', async response =>{
      try {
       console.succ(await response.json());
      } catch (error) {
        //
        // console.error(error);
      }
  
  
  
  });

and it sure get one.

the problem is ---> that when I'm deploying it to the firebase cloud functions server,

firebase deploy --only functions

and then triggering the function --> I'm getting a json that look like this:

{ success: false, message: 'You look like a robot.' } 

But when deploying the same code to my local host like so

firebase serve --only functions

and then triggering the function --> I'm not detected as a robot and getting the json with a successful result --> and with that data that the clicking of a button supposed to fetch.

this is so weird, I'm trying to think that there's a connections between the firebase cloud functions and reCAPTCHA, because both are a google services but, its not seem's reasonable for it to be true.

that being said, what could be the reason for this? all that change is the environment that the code runs from. do you have any idea why this is happening? and how to solve it of course.

Answer 1

Since your function runs properly locally, it's almost certainly not the function itself.

Sites take a variety of different approaches to detect bots, one of which is blocking traffic from known data centers like Google Cloud's. Using a residential IP proxy like those provided by BrightData will probably circumvent this.

Answer 2

I'm facing the same issue while using Puppeteer in Firebase Cloud Functions.

I'm using a residential IP proxy with the following set of packages puppeteer-extra , puppeteer-extra-plugin-stealth , puppeteer-extra-plugin-anonymize-ua', and user-agents`.

On localhost, all is working as expected while running Puppeteer in firebase Cloud Functions I'm getting a 404 response from the requested URL. So there must be some difference.