How KCL application is integrated to Kinesis Connector to emit data on S3

Question

Every Kinesis application must include these three components:

1. IRecordProcessor interface

2. A factory for the class that implements the IRecordProcessor interface

3. Code that initializes the application and creates the worker

Now i understand here that once we configure the producer to add records to Kinesis stream . Then KCL application can read records from Kinesis stream with below implementation of processRecords. then this processRecords method should have some way to pass it to S3 for final storage with connector library.

public void processRecords(List records, IRecordProcessorCheckpointer checkpointer)

Query: How should i call connector library from processRecords of KCL application to store the data records on S3?

I went through the link which shows a sample Kinesis application https://github.com/aws/aws-sdk-java/blob/master/src/samples/AmazonKinesisApplication/SampleRecordProcessor.java

in the above link i am pasting method snippet as below.

private void processRecordsWithRetries(List<Record> records) {
        for (Record record : records) {
            boolean processedSuccessfully = false;
            String data = null;
            for (int i = 0; i < NUM_RETRIES; i++) {
                try {
                    // For this app, we interpret the payload as UTF-8 chars.
                    data = decoder.decode(record.getData()).toString();
                    LOG.info(record.getSequenceNumber() + ", " + record.getPartitionKey() + ", " + data);
            //
                    // Logic to process record goes here.
                    //
                    processedSuccessfully = true;
                    break;
                } catch (CharacterCodingException e) {
                    LOG.error("Malformed data: " + data, e);
                    break;
                } catch (Throwable t) {
                    LOG.warn("Caught throwable while processing record " + record, t);
                }

                // backoff if we encounter an exception.
                try {
                    Thread.sleep(BACKOFF_TIME_IN_MILLIS);
                } catch (InterruptedException e) {
                    LOG.debug("Interrupted sleep", e);
                }
            }

            if (!processedSuccessfully) {
                LOG.error("Couldn't process record " + record + ". Skipping the record.");
            }
        }
    }

In the above code when we say " Logic to process record goes here." (please see code above) Here my requirement is to put data on s3 . I understand that we have connector library which does this however i am not able to visualize how to call connector library at this point onwards ?please suggest

Answer 1

您应该尝试kinesis连接器库，它提供了您所需的样本： https : //github.com/awslabs/amazon-kinesis-connectors 。