How do I give access to 1000's (100,000) of remote deployed devices (customer site servers) to upload logs to aws kinesis?

Question

There are close to 100,000 devices that are generating logs (total of 10-20 TB a day) which I would like them to directly upload to kinesis. How do I control access? IAM only lets me create a max of 1000 users per account (I know we can request user limit increase), but would like to know what is a better way to do this.

One requirement is, I would like to be able to grant/revoke access to kinesis per device.

Answer 1

Since you have IoT Core already, I think that I would first try to leverage it for logging. This will let you take advantage of the certificate-based authorization that's built-in to IoT core, and I know that you can hook an IoT topic into a Kinesis stream.

If you feel that this would be too much volume (and perhaps too expensive based on the number of messages and rules), then I'd provide my devices with temporary security credentials that let them write to Kinesis and nothing else.

You would generate these credentials on a per-device basis (as far as I can tell, there are no quotas on the number of credentials per account), using a scheduled job, either in Lambda or on ECS. This job would iterate through your devices and generate a set of credentials for each. It would then either publish these credentials to the device via IoT Core, or update the device shadow.

The device could then use these credentials to create a Kinesis client to publish log messages. Your client would have to create a new client whenever it receives new credentials.

As an alternative, if your devices maintain logfiles internally, you could use a similar approach to trigger uploading those files to S3. In that case, rather than publishing temporary credentials, the scheduled task would publish a pre-signed URL for each device. It would publish the URL to the device, and the device would use that to upload its accumulated logs. Then you'd need something to do something with the files on S3.