简体   繁体   English

需要有关如何使用 Cassandra Kafka 连接器的帮助

[英]Need help on how to use Cassandra Kafka Connector

I need help on this github code: https://github.com/castagna/kafka-connect-cassandra我需要有关此 github 代码的帮助: https://github.com/castagna/kafka-connect-cassandra

The goal of this code is to receive datas from Apache Kafka, and then add them into Cassandra database automatically (It is called a connector).这段代码的目标是从 Apache Kafka 接收数据,然后将它们自动添加到 Cassandra 数据库中(称为连接器)。

I was able to compile it normally, but there is no much infos on how to use it.我能够正常编译它,但没有太多关于如何使用它的信息。

My only concern now is to know what is the format of messages that I have to send to this connector using a JSON format, so he can interpret it and add to Cassandra database.我现在唯一关心的是知道我必须使用 JSON 格式发送到此连接器的消息格式是什么,以便他可以解释它并添加到 Cassandra 数据库。

The function responsible to put datas to Cassandra DB: function 负责将数据放入 Cassandra DB:

@Override
public void put(Collection<SinkRecord> records) {
    if (records.isEmpty()) {
        return;
    }
    final SinkRecord first = records.iterator().next();
    final int recordsCount = records.size();
    log.trace("Received {} records. First record kafka coordinates:({}-{}-{}). Writing them to Cassandra...", recordsCount, first.topic(), first.kafkaPartition(), first.kafkaOffset());
    try {
        writer.write(records);
    } catch (Exception e) {
        log.warn("Write of {} records failed, remainingRetries={}", records.size(), remainingRetries, e);
        if (remainingRetries == 0) {
            throw new ConnectException(e);
        } else {
            writer.closeQuietly();
            initWriter();
            remainingRetries--;
            // context.timeout(10000); // TODO: make this a configurable option
            throw new RetriableException(e);
        }
    }
    remainingRetries = config.maxRetries;
}

What I have tried:我试过的:

Keyspace and table:键空间和表:

CREATE KEYSPACE demo WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 3};
use demo;

create table products (id int, product varchar, price float, PRIMARY KEY (id));

Format of message used (JSON):使用的消息格式(JSON):

{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"}, {"type":"string","optional":true,"field":"product"}, {"type":"int32","optional":false,"field":"price"}],"optional":false,"name":"orders"},"payload":{"id":1, "product": "test", "price":120}}

And finally config file (cassandra-sink.properties):最后是配置文件(cassandra-sink.properties):

name=cassandra-sink-products
connector.class=com.github.castagna.kafka.connect.cassandra.CassandraSinkConnector
tasks.max=1
topics=testTopic

cassandra.host=127.0.0.1
cassandra.keyspace.name=demo

auto.create=true
auto.evolve=true

Your help will be appreciated, thank you.您的帮助将不胜感激,谢谢。

That connector is pretty old - you want to get a currently-maintained connector eg那个连接器已经很老了——你想得到一个当前维护的连接器,例如

To use Kafka Connect you specify a configuration file and pass this as an argument to Kafka Connect when you run it (or pass it to its REST API).要使用 Kafka Connect,您需要指定一个配置文件,并在运行时将其作为参数传递给 Kafka Connect(或将其传递给其 REST API)。

I'd suggest reviewing this talk on what Kafka Connect is and how to use it, and this quickstart for examples on how to run it.我建议查看有关 Kafka Connect 是什么以及如何使用它的演讲,以及有关如何运行它的示例的快速入门 You can find more information about Kafka Connect here .您可以在此处找到有关 Kafka Connect 的更多信息。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM