Connecting AWS-Neptune from EC2 with Signature version 4 signing

Question

I'm having difficulties connecting to Amazon Neptune Cluster from EC2 instance using any of sample applications with my IAM user :

1. https://docs.aws.amazon.com/neptune/latest/userguide/iam-auth-connecting-gremlin-java.html
2. https://docs.aws.amazon.com/neptune/latest/userguide/iam-auth-connecting-python.html .

I believe the SG is correctly setup, because if I disable IAM db authentication, my connection succeeds.

I'm providing my IAM credentials with env variables:

export AWS_ACCESS_KEY_ID=access key
export AWS_SECRET_ACCESS_KEY=secret
export SERVICE_REGION=region

My IAM User has policies applied: arn:aws:iam::aws:policy/AmazonRDSFullAccess, and arn:aws:iam::aws:policy/NeptuneFullAccess

Calling sample java app (from link above) results in 403 Forbidden error:

07:54:56.274 [gremlin-driver-loop-1] ERROR o.a.t.g.d.Handler$GremlinResponseHandler - Could not process the response
io.netty.handler.codec.http.websocketx.WebSocketClientHandshakeException: Invalid handshake response getStatus: 403 Forbidden
    at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:272)
    at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker.finishHandshake(WebSocketClientHandshaker.java:302)
    at io.netty.handler.codec.http.websocketx.WebSocketClientProtocolHandshakeHandler.channelRead(WebSocketClientProtocolHandshakeHandler.java:93)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
    at io.netty.handler.codec.http.websocketx.extensions.WebSocketClientExtensionHandler.channelRead(WebSocketClientExtensionHandler.java:128)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1504)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1253)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1300)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.base/java.lang.Thread.run(Thread.java:829)

Any thoughts on how to debug this appreciated. Searching SO didn't help.

UPDATE:

After pointing out, that my credentials could be incorrect, I found this link https://docs.aws.amazon.com/neptune/latest/userguide/manage-console-iam-user.html , went through it, updated my user with policies from the link:

NeptuneFullAccess

NeptuneConsoleFullAccess

NeptuneServiceLinked

AmazonVPCFullAccess

I still get error 403 using awscurl:

{"code":"AccessDeniedException","requestId":"xxx","detailedMessage":"You do not have sufficient access to perform this action."}

I must be missing something obvious here

Answer 1

Any easy way to validate that your IAM credentials are working correctly is to use awscurl [1]. awscurl will not pull directly from the IAM role associated with the EC2 instance, so you'll need to explicitly define the environment variables [2] for AWS_ACCESS_KEY_ID, AWS_SECRET_KEY, and AWS_SESSION_TOKEN (if using temp credentials) or define these same values as parameters in the awscurl call.

awscurl https://<neptune_endpoint>:8182/status --service neptune-db --region <region_name>

This should return the status of the cluster if your credentials are defined correctly.

[1] https://github.com/okigan/awscurl

[2] Script to pull IAM role from EC2 instance or ECS container and define appropriate environment variables: https://docs.aws.amazon.com/neptune/latest/userguide/iam-auth-temporary-credentials.html#iam-auth-temporary-credentials-ec2

Answer 2

You could be missing the neptune-db:* IAM permission for the credentials. This permission is required for connection to Neptune clusters with IAM based authentication enabled.

Please find the permission mentioned in Neptune documentation .