使用 Kinesis Firehose 将数据写入 S3 时出现问题，所有记录都是 404 条消息

Question

I've set up a Kinesis Firehose stream which successfully runs the "test" in the AWS console. My code writing to the stream looks like this:

return graphqlHTTP(
(request: Object, response: Object, params: Object): Object => {
  const firehoseConfig: Object = {
    region: config.get('awsRegion'),
    credentials = {
      accessKeyId: config.get('awsAccessKey'),
      secretAccessKey: config.get('secretAccessKey'),
    }
  }
  // Init Kinesis Firehose
  const firehose = new AWS.Firehose(firehoseConfig)
  // Prep response object for sending
  const stringifiedResponse = JSON.stringify(response)
  // Send to Kinesis Firehose stream
  const firehoseParams = {
    DeliveryStreamName: 'test-stream',
    Record: {
      Data: Buffer.from(stringifiedResponse),
    },
  }
  firehose.putRecord(firehoseParams, (err: Object, data: Object) => {
    // eslint-disable-next-line no-console
    if (err) console.log('FIREHOSE ERROR: ', err, err.stack)
    // eslint-disable-next-line no-console
    else console.log(data)
  })

Records are writing to the stream when the endpoint is hit, however all records look like this:

 {
  "status": 404,
  "message": "Not Found",
  "header": {
    "x-frame-options": "SAMEORIGIN",
    "strict-transport-security": "max-age=86400",
    "x-download-options": "noopen",
    "x-content-type-options": "nosniff",
    "x-xss-protection": "1; mode=block",
    "vary": "Accept-Encoding, Origin",
    "cache-control": "max-age=60, s-maxage=60",
    "access-control-allow-origin": "<redacted internal site>",
    "access-control-allow-credentials": "true",
    "access-control-expose-headers": "content-length,etag",
    "x-ratelimit-remaining": "996",
    "x-ratelimit-reset": "1571372307",
    "x-ratelimit-limit": "1000"
  }
} {
  "status": 404,
  "message": "Not Found",
  "header": {
    "x-frame-options": "SAMEORIGIN",
    "strict-transport-security": "max-age=86400",
    "x-download-options": "noopen",
    "x-content-type-options": "nosniff",
    "x-xss-protection": "1; mode=block",
    "vary": "Accept-Encoding, Origin",
    "cache-control": "max-age=60, s-maxage=60",
    "access-control-allow-origin": "<redacted internal site>",
    "access-control-allow-credentials": "true",
    "access-control-expose-headers": "content-length,etag",
    "x-ratelimit-remaining": "995",
    "x-ratelimit-reset": "1571372307",
    "x-ratelimit-limit": "1000"
  }
}

What I'm trying to do is have this Firehose send the response to S3 so that I can later run queries on it with Athena.

Answer 1

It turns out that a 404 from Kinesis, per the docs , indicates a malformed query. In this context it means that the Data being provided in Record isn't valid JSON. The response object, which here is provided by Koa/ koa-graphql , is a large object with circular references, and somewhere therein is invalid JSON or JSON that AWS considers to be invalid.