堆栈内存溢出
  简体   繁体   English

将 Apache Kafka 连接到数据库

[英]Connect Apache Kafka to a database

 原文  2017-09-11 11:14:00  6   3    apache-kafka/ apache-kafka-connect

问题描述

I am trying to create a streaming/big-data application with PostgreSQL, Apache Kafka (to send data) and Apache Spark Streaming (to process the data).我正在尝试使用 PostgreSQL、Apache Kafka（发送数据）和 Apache Spark Streaming（处理数据）创建流/大数据应用程序。

Following this document , I first try to connect Kafka to a Sqlite database.按照本文档，我首先尝试将 Kafka 连接到 Sqlite 数据库。 But when I run the connector with the command但是当我使用命令运行连接器时

$ ./bin/connect-standalone ./etc/schema-registry/connect-avro-standalone.properties ./etc/kafka-connect-jdbc/source-quickstart-sqlite.properties

I got the errors below, before the process shutdown.在进程关闭之前，我收到了以下错误。 I pass the correct path to my database, I verified.我将正确的路径传递给我的数据库，我验证了。

How can I connect and work the connection between my database & Kafka ?如何连接和工作我的数据库和 Kafka 之间的连接？

2017-09-11 12:00:17,305] INFO jetty-9.2.15.v20160210 (org.eclipse.jetty.server.Server:327)
Sep 11, 2017 12:00:17 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

[2017-09-11 12:00:17,618] INFO Started o.e.j.s.ServletContextHandler@1a2d50d5{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744)
[2017-09-11 12:00:17,623] WARN FAILED ServerConnector@3e37a981{HTTP/1.1}{0.0.0.0:8083}: java.net.BindException: Address already in use (org.eclipse.jetty.util.component.AbstractLifeCycle:212)
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.server.Server.doStart(Server.java:366)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:145)
    at org.apache.kafka.connect.runtime.Connect.start(Connect.java:53)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82)
[2017-09-11 12:00:17,624] WARN FAILED org.eclipse.jetty.server.Server@46eb180e: java.net.BindException: Address already in use (org.eclipse.jetty.util.component.AbstractLifeCycle:212)
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.server.Server.doStart(Server.java:366)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:145)
    at org.apache.kafka.connect.runtime.Connect.start(Connect.java:53)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82)
[2017-09-11 12:00:17,624] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:100)
org.apache.kafka.connect.errors.ConnectException: Unable to start REST server
    at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:147)
    at org.apache.kafka.connect.runtime.Connect.start(Connect.java:53)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82)
Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.server.Server.doStart(Server.java:366)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:145)
    ... 2 more

3 个解决方案

解决方案1
 0  2017-09-12 06:42:08

WARN FAILED ServerConnector@3e37a981{HTTP/1.1}{0.0.0.0:8083}: java.net.BindException: Address already in use (org.eclipse.jetty.util.component.AbstractLifeCycle:212)
java.net.BindException: Address already in use

This means that something else is already using port 8083. 这意味着其他端口已经在使用端口8083。

Are you using Confluent Platform ? 您正在使用Confluent Platform吗？ This is a great way to get started with Apache Kafka, and gives you an easy way to bring up the stack ( confluent start ) as well as configure Kafka Connect. 这是开始使用Apache Kafka的好方法，并且为您提供了一个简便的方法来confluent start堆栈（ confluent start ）以及配置Kafka Connect。 This blog series shows it in action: 这个博客系列展示了它的实际作用：

BTW you might want to check out Kafka Streams too, as a way of doing stream processing within Kafka itself. 顺便说一句，您可能也想签出Kafka Streams ，作为在Kafka本身内部进行流处理的一种方式。

解决方案2
 0  2019-02-21 03:34:07

I commented out the listeners=http://0.0.0.0:8081 property in the schema-registry.properties file and the issue was resolved. 我注释掉了schema-registry.properties文件中的listeners=http://0.0.0.0:8081属性，此问题已解决。

Your port number is probably 8083 , though. 但是，您的端口号可能是8083 。

解决方案3
 0  2021-11-09 18:57:15

请注意，在Kafka 3.0 ，属性rest.port=<port>已弃用，现在您必须使用listeners=<http:port>

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 是否可以将Apache Kafka连接到Azure SQL数据库？ - Is it possible to connect Apache Kafka to Azure SQL Database? Apache Kafka与Springboot连接 - Apache Kafka Connect With Springboot Kafka与Cassandra数据库连接 - Kafka connect with Cassandra database 将kafka连接到jdbc数据库 - Connect kafka to jdbc database Teradata 和 Apache Kafka 使用 Kafka 连接 - Teradata and Apache Kafka using Kafka connect 带有Kafka Connect和Apache Kafka的MS SQL CDC - MS SQL CDC with Kafka Connect and Apache Kafka Apache Kafka Connect任务重启 - Apache Kafka Connect Task Restart 将JMS客户端连接到Apache Kafka - Connect JMS client to Apache Kafka Confluent JDBC 与 Apache Kafka 连接 - Confluent JDBC connect with Apache Kafka 无法连接和描述 Kafka 集群。 Apache 卡夫卡连接 - Failed to connect to and describe Kafka cluster. Apache kafka connect
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM