对外部 API 的相同请求适用于 go、postman、curl，但不适用于 node.js

Question

I have a strange problem while performing request to external API. Basically I got 403 error response for the HTTP request, but... only when calling it from node environment. I tried with curl through terminal and it's OK. I also tried from Postman and native go with net/http package and everything is just fine, I got 200 response. But no matter what tool I use in node.js (request, axios, unirest, got) I can't make it. I would love to share entire request with you, but unfortunately I can't :(

All I can show you is some response details, when I call it from node.js environment.

These are headers sent by the server when I got 403 :

  headers: {
    server: 'AkamaiNetStorage',
    'content-length': '228',
    'content-type': 'application/json',
    'x-edge-error': 'halt',
    'cache-control': 'max-age=122',
    date: 'Thu, 21 Oct 2021 16:17:41 GMT',
    connection: 'close'
  },

And the part of response data looks like this:

  data: {
    edge_error: 'halt',
    ref_id: '18.9d6656b8.1634833061.4b9f14b',
  }

I am aware that question is strange and you probably won't be able to help me with not so many details, but I will try anyway. So If anyone have idea what might help, feel free to propose an answer.

Best regards ;)

Answer 1

This is likely due to TLS fingerprinting. Akamai is able to detect that you're using Node.JS based on information gathered during the TLS handshake (cipher suites, etc.) when you send a request. The website must have set a rule through Akamai to block Node.

Try something like this:

const tls = require('tls');
const https = require('https');

const defaultCiphers = tls.DEFAULT_CIPHERS.split(':');
const shuffledCiphers = [
    defaultCiphers[0],
    // Swap the 2nd & 3rd ciphers:
    defaultCiphers[2],
    defaultCiphers[1],
    ...defaultCiphers.slice(3)
].join(':');

request = require('https').get('https://google.com', {
    ciphers: shuffledCiphers
}).on('response', (res) => {
    console.log(res.statusCode);
});
// source: https://httptoolkit.tech/blog/tls-fingerprinting-node-js/

More information about this can be found here: https://httptoolkit.tech/blog/tls-fingerprinting-node-js/