消费者的 Amazon Kinesis KCL 客户端在 .NET 中不工作

Question

please help. I am having problems setting up the Consumer for Kinesis data stream in .NET console application.

I have done everything according to the documentation, but I still get a blank console screen whenever I run the consumer. The producer works fine so far and the AWS credentials are working.

1. I have JDK on my system all well configured (not new to Java development)
2. I have all the necessary policies attached to my IAM user
3. I can see that the producer can programmatically create streams, desc streams etc with the same AWS credentials

I can hit the breakpoint in Program when creating the KclProcess but I cannot hit any breakpoint inside the KinesisTest class below

As for the consumer, I've created a class Program.cs like below:

class Program  
{

    public static void Main(string[] args)
    {
        //added these lines after trying everything
        Environment.SetEnvironmentVariable("AWS_ACCESS_KEY_ID", "***");
        Environment.SetEnvironmentVariable("AWS_SECRET_ACCESS_KEY", "***");
        Environment.SetEnvironmentVariable("AWS_REGION", "us-east-1");

        try
        {
            KclProcess.Create(new KinesisTest()).Run();
        }
        catch (Exception e)
        {
            Console.Error.WriteLine("ERROR: " + e);
        }
    }

}

and another class

public class KinesisTest: IRecordProcessor
{

    private static readonly TimeSpan Backoff = TimeSpan.FromSeconds(3);
    private static readonly TimeSpan CheckpointInterval = TimeSpan.FromMinutes(1);
    private static readonly int NumRetries = 10;

    /// <value>The shard ID on which this record processor is working.</value>
    private string _kinesisShardId;

    private DateTime _nextCheckpointTime = DateTime.UtcNow;


    public void Initialize(InitializationInput input)
    {
        Console.Error.WriteLine("Initializing record processor for shard: " + input.ShardId);
        this._kinesisShardId = input.ShardId;
    }

    public void ProcessRecords(ProcessRecordsInput input)
    {
        Console.Error.WriteLine("Processing " + input.Records.Count + " records from " + _kinesisShardId);
        ProcessRecordsWithRetries(input.Records);

        // Checkpoint once every checkpoint interval.
        if (DateTime.UtcNow >= _nextCheckpointTime)
        {
            Checkpoint(input.Checkpointer);
            _nextCheckpointTime = DateTime.UtcNow + CheckpointInterval;
        }
    }

    public void Shutdown(ShutdownInput input)
    {
        Console.Error.WriteLine("Shutting down record processor for shard: " + _kinesisShardId);
        // Checkpoint after reaching end of shard, so we can start processing data from child shards.
        if (input.Reason == ShutdownReason.TERMINATE)
        {
            Checkpoint(input.Checkpointer);
        }
    }

    private void ProcessRecordsWithRetries(List<Record> records)
    {
        foreach (Record rec in records)
        {
            bool processedSuccessfully = false;
            string data = null;
            for (int i = 0; i < NumRetries; ++i)
            {
                try
                {
                    data = System.Text.Encoding.UTF8.GetString(rec.Data);

                    Console.Error.WriteLine( String.Format("Retrieved record:\n\tpartition key = {0},\n\tsequence number = {1},\n\tdata = {2}", rec.PartitionKey, rec.SequenceNumber, data));

                    // Your own logic to process a record goes here.

                    processedSuccessfully = true;
                    break;
                }
                catch (Exception e)
                {
                    Console.Error.WriteLine("Exception processing record data: " + data, e);
                }

                //Back off before retrying upon an exception.
                Thread.Sleep(Backoff);
            }

            if (!processedSuccessfully)
            {
                Console.Error.WriteLine("Couldn't process record " + rec + ". Skipping the record.");
            }
        }
    }

    private void Checkpoint(Checkpointer checkpointer)
    {
        Console.Error.WriteLine("Checkpointing shard " + _kinesisShardId);

        checkpointer.Checkpoint(RetryingCheckpointErrorHandler.Create(NumRetries, Backoff));
    }
}

and lastly the kcl.properties file:

executableName = dotnet KinesisTest.dll

streamName = testStream

applicationName = KinesisTest

AWSCredentialsProvider = DefaultAWSCredentialsProviderChain

processingLanguage = C#

initialPositionInStream = TRIM_HORIZON

regionName = us-east-1

maxRecords = 5000

idleTimeBetweenReadsInMillis = 1000

# failoverTimeMillis = 10000
# workerId =
# shardSyncIntervalMillis = 60000
# callProcessRecordsEvenForEmptyRecordList = false
# parentShardPollIntervalMillis = 10000
# cleanupLeasesUponShardCompletion = true
# taskBackoffTimeMillis = 500
# metricsBufferTimeMillis = 10000
# metricsMaxQueueSize = 10000
# validateSequenceNumberBeforeCheckpointing = true
# maxActiveThreads = 0

Please let me know if I'm doing something wrong.

I was expecting to see the Consumer processing the data from the stream but it's just an empty console

Answer 1

While I never found an answer to this question, I found a better and proper way to do this using lambdas.

My final setup involved using SNS to get the messages/events from Kinesis and then fanning out the message to any subscribers (In my case, SQS queues) which then provide the queued messages to lambdas.

Something like this:

                             SQS Queue ----------------> Lambda
                             ^
                            /
                           /
                          /
KINESIS STREAM ------->SNS-------->SQS Queue ------> Lambda
                          \
                           \
                            \
                             >SQS Queue ----------------> Lambda

You might ask why not Kinesis to SQS directly? or why not SNS to Lambda directly?

First question's answer is that Kinesis isn't build for multiple reads. SNS allows you to read once and fan it out to as many subscribers as you need. If you have say 20 listeners who are waiting to act on an event, you can't plug them all into Kinesis. So go SNS.

Second question's answer is that SNS sometimes can lose messages and not deliver or trigger the lambdas (very true). However (speculated based on experience) this probability reduces if you connect it to SQS, it almost always lands in the queue. The queues can then trigger the lambdas.. They have only one job anyway.

So I hope this helps someone.

Answer 2

You have to run this by Bootstrap project. Bootstrap project run Java program which run your project. You can find ready Bootstrap project in KCL for .NET repo.

https://github.com/awslabs/amazon-kinesis-client-net/tree/master/Bootstrap

Perhaps you will have to do some changes in this project like a set "pom.xml" location or something else but you can debug this project to find out if everything started correctly.