Using AWS Kinesis with localstack and Apache Flink ingress

Question

I have localstack running a mock Kinesis stream within my Docker Compose at http://localstack:4645

I am using the Flink Kinesis ingress io type to attempt to connect to the stream.

https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2/docs/modules/io/aws-kinesis/

Following the above I created

kind: io.statefun.kinesis.v1/ingress
spec:
  id: com/sentiment
  awsRegion:
    type: specific
    id: us-east-1
  awsCredentials:
    type: custom-endpoint
    endpoint: https://localhost:4566
    id: us-east-1
    accessKeyId: key
    secretAccessKey: secret
  startupPosition:
    type: latest
  streams:
    - stream: customer-details
      valueType: namespace/SentimentEvent
      targets:
        - namespace/sentiment
  clientConfigProperties:
    - SocketTimeout: 9999
    - MaxConnections: 15

I then start up my Docker Compose file which contains the Flink StateFun image but I get this error within my logs:

Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException: Invalid AWS credential type: custom-endpoint; valid values are [default, basic, profile] (through reference chain: org.apache.flink.statefun.flink.io.kinesis.binders.ingress.v1.RoutableKinesisIngressSpec$Builder["awsCredentials"])

It's confusing as the docs say:

Custom Endpoint #
Connects to an AWS region through a non-standard AWS service endpoint. This is typically used only for development and testing purposes.

awsCredentials:
  type: custom-endpoint
  endpoint: https://localhost:4567
  id: us-west-1

But it doesn't seem to be valid configuration.

I can access the Localstack stream fine with the awslocal tool and my other applications within my Docker Compose so I know that's not the issue.

I can't see seem to figure out the issue.

Answer 1

Turns out endpoint needs to go under the awsRegion section in the config.

  awsRegion:
    type: custom-endpoint
    id: us-east-1
    endpoint: https://localstack:4566