在nodejs环境的aws lambda模块中提出请求时，将为两个特定域返回403

Question

When request is made in aws lambda module of nodejs environment, 403 is returned for two specific domains. However, calling that domain alone will not return 403. The same applies to the request module, not the cralwer module.

Executing the same logic locally returns all of the responses back to 200 normally.

This is the source uploaded to the lambda.

const Crawler = require('crawler');
const urls = [
  'http://www.ddengle.com',
  'http://www.cointalk.co.kr',
  'http://www.chaintalk.io',
  'http://www.coinpan.com',
  'http://www.hozaebox.com',
  'https://gall.dcinside.com/board/lists?id=bitcoins',
  'https://gall.dcinside.com/mgallery/board/lists?id=coin',
];

exports.handler = async (event) => {
  return new Promise(async (resolve) => {
    const crawler = new Crawler({
      maxConnections: 10,
      jQuery: 'whacko',
      callback(err, res, done) {
        if (err) throw err;
        const hostname = res.request.uri.hostname;

        if (res.statusCode === 200) {
          console.log(hostname);
        } else console.log(`[crawler] ${hostname} statusCode ${res.statusCode}`);

        done();
      },
    });

    crawler.on('drain', () => {
      resolve();
    });

    urls.forEach((e) => {
      crawler.queue([{
        headers: {
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36',
        },
        uri: e,
      }]);
    });
  });
};

For my purposes it is simply a request call, so all values ​​should be answered with 200. However, running the test on the lambda will return 403 unconditionally for either www.ddengle.com or coinpan.com.

START RequestId: f3cc4977-11af-4ab1-9556-9b778efd1f72 Version: $LATEST
2019-08-23T08:13:36.593Z    f3cc4977-11af-4ab1-9556-9b778efd1f72    INFO    www.chaintalk.io
2019-08-23T08:13:36.811Z    f3cc4977-11af-4ab1-9556-9b778efd1f72    INFO    [crawler] www.ddengle.com statusCode 403
2019-08-23T08:13:37.170Z    f3cc4977-11af-4ab1-9556-9b778efd1f72    INFO    www.hozaebox.com
2019-08-23T08:13:37.454Z    f3cc4977-11af-4ab1-9556-9b778efd1f72    INFO    gall.dcinside.com
2019-08-23T08:13:37.873Z    f3cc4977-11af-4ab1-9556-9b778efd1f72    INFO    gall.dcinside.com
2019-08-23T08:13:38.391Z    f3cc4977-11af-4ab1-9556-9b778efd1f72    INFO    www.cointalk.co.kr
2019-08-23T08:13:39.153Z    f3cc4977-11af-4ab1-9556-9b778efd1f72    INFO    coinpan.com
END RequestId: f3cc4977-11af-4ab1-9556-9b778efd1f72

Answer 1

Kindly move your Lambda into a VPC, then Lambda request is made from a specific IP. I am assuming that when Lambda runs outside a VPC (Default behavior), there is no associated IP to it and it might be blocked by the website which you are trying to crawl.

Answer 2

Domain ddengle.com seems to be protected by cloudflare and only accessible through https .

curl -I http://www.ddengle.com
HTTP/1.1 301 Moved Permanently
Date: Fri, 23 Aug 2019 08:31:50 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Fri, 23 Aug 2019 09:31:50 GMT
Location: https://www.ddengle.com/
X-Content-Type-Options: nosniff
Server: cloudflare

It can be country block, WAF, browser integrity check ...

Full list here: https://community.cloudflare.com/t/community-tip-fixing-error-403-forbidden/53308