AWS IAM role expiry

Question

On expiry of IAM roles:

• What is the logic for IAM roles to expire, when you need them for longer. I really cannot see it.
• How do you when processing, get the an extension of the IAM role without some serious failure?
  • I did the IAM course but felt that was not well addressed.
  • Eg a long running AWS EMR Spark data pipeline on a massive scale for cross account access?
    • If the Spark data pipeline has finished the Stage of reading from S3 and that role expires, may be that does not matter if you do not save subsequently to S3.

Answer 1

Instance roles such as those used in EMR are renewed automatically:

The application is granted the permissions for the actions and resources that you've defined for the role through the security credentials associated with the role. These security credentials are temporary and we rotate them automatically . We make new credentials available at least five minutes before the expiration of the old credentials.

Answer 2

Look in the AWS SDK for com.amazonaws.auth.InstanceProfileCredentialsProvider ; this is called by the clients to get the IAM Credentials. It spawns a thread com.amazonaws.auth.EC2CredentialsFetcher which does HTTP requests to the special 169.xxx http server which provides these details. Every spark worker creating an s3 client (or s3a, on ASF builds) will instantiate an InstanceProfileCredentialsProvider, after which everything will "just work"

IAM roles used to always expire after 1h; any job lasting 65+ minutes would have triggered a refresh.

try it and see.