堆栈内存溢出
  简体   繁体   English

Cygnus 不会将数据持久化到 PostSQL 中

[英]Cygnus does not persist data into PostSQL

 原文  2022-03-10 17:19:36  2   1    postgresql/ fiware/ fiware-cygnus

问题描述

I am using the following tutorial to persist data in a PostGresQL database via Cygnus.我正在使用以下教程通过 Cygnus 将数据保存在 PostGresQL 数据库中。

https://documenter.getpostman.com/view/513743/RWEcR2DC https://documenter.getpostman.com/view/513743/RWEcR2DC

When using a MySQL database with Cygnus everything works fine as you can see below:当 MySQL 数据库与 Cygnus 一起使用时，一切正常，如下所示：

mysql> SHOW tables FROM openiot;
+-------------------------------------------------------+
| Tables_in_openiot                                     |
+-------------------------------------------------------+
| urn_ngsi-ld_TemperatureSensor_11401_TemperatureSensor |
| urn_ngsi-ld_TemperatureSensor_12435_TemperatureSensor |
| urn_ngsi-ld_TemperatureSensor_13477_TemperatureSensor |
| urn_ngsi-ld_TemperatureSensor_13828_TemperatureSensor |
| urn_ngsi-ld_TemperatureSensor_15205_TemperatureSensor |
| urn_ngsi-ld_TemperatureSensor_1546_TemperatureSensor  |
+-------------------------------------------------------+
109 rows in set (0.00 sec)

But when using PostGres as backend, the NGSIPostgreSQLSink does not create tables.但是当使用 PostGres 作为后端时，NGSIPostgreSQLSink 不会创建表。

Here is my setup on docker:这是我在 docker 上的设置：

  postgres:
    image: postgres:latest
    hostname: postgres
    expose:
      - "5432"
    ports:
      - "5432:5432"
    environment:
      - "POSTGRES_PASSWORD=password"
      - "POSTGRES_USER=postgres"
      - "POSTGRES_DB=postgres"
    volumes:
      - data-postgres:/var/lib/postgresql/data
    
  cygnus:
    image: fiware/cygnus-ngsi:latest
    hostname: cygnus
    depends_on:
        - postgres
    expose:
        - "5080"
    ports:
        - "5055:5055"
        - "5080:5080"
    environment:
        - "CYGNUS_POSTGRESQL_HOST=postgres"
        - "CYGNUS_POSTGRESQL_PORT=5432"
        - "CYGNUS_POSTGRESQL_USER=postgres"
        - "CYGNUS_POSTGRESQL_PASS=password"
        - "CYGNUS_POSTGRESQL_ENABLE_CACHE=true"
        - "CYGNUS_POSTGRESQL_SERVICE_PORT=5055"
        - "CYGNUS_LOG_LEVEL=DEBUG"
        - "CYGNUS_API_PORT=5080"
        - "CYGNUS_SERVICE_PORT=5055"

If I list the schemas with a PostGres client there is no openiot schema:如果我使用 PostGres 客户端列出模式，则没有 openiot 模式：

postgres=# \dn
  List of schemas
  Name  |  Owner
--------+----------
 public | postgres
(1 row)

I also tried to mount the following agent.conf and cygnus_instance.conf files in docker:我还尝试在docker中挂载了以下agent.conf和cygnus_instance.conf文件：

docker docker

volumes:
   - ./conf/cygnus:/opt/apache-flume/conf/agent:ro

agent.conf代理配置文件

cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = postgresql-sink
cygnus-ngsi.channels = postgresql-channel


cygnus-ngsi.sources.http-source.channels = postgresql-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
cygnus-ngsi.sources.http-source.interceptors.nmi.type = com.telefonica.iot.cygnus.interceptors.NGSINameMappingsInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.nmi.name_mappings_conf_file = /usr/cygnus/conf/name_mappings.conf


cygnus-ngsi.sinks.postgresql-sink.channel = postgresql-channel
cygnus-ngsi.sinks.postgresql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink
cygnus-ngsi.sinks.postgresql-sink.enable_encoding = false
cygnus-ngsi.sinks.postgresql-sink.enable_name_mappings = false
cygnus-ngsi.sinks.postgresql-sink.enable_grouping = false
cygnus-ngsi.sinks.postgresql-sink.enable_lowercase = false
cygnus-ngsi.sinks.postgresql-sink.postgresql_host = postgres
cygnus-ngsi.sinks.postgresql-sink.postgresql_port = 5432
cygnus-ngsi.sinks.postgresql-sink.postgresql_database = postgres
cygnus-ngsi.sinks.postgresql-sink.postgresql_username = postgres
cygnus-ngsi.sinks.postgresql-sink.postgresql_password = password
cygnus-ngsi.sinks.postgresql-sink.attr_persistence = column
cygnus-ngsi.sinks.postgresql-sink.data_model = by-service-path
cygnus-ngsi.sinks.postgresql-sink.batch_size = 100
cygnus-ngsi.sinks.postgresql-sink.batch_timeout = 30
cygnus-ngsi.sinks.postgresql-sink.batch_ttl = 10
cygnus-ngsi.sinks.postgresql-sink.persist_errors = true


cygnus-ngsi.channels.postgresql-channel.type = memory
cygnus-ngsi.channels.postgresql-channel.capacity = 100000
cygnus-ngsi.channels.postgresql-channel.transactionCapacity = 10000

cygnus_instance.conf cygnus_instance.conf

CYGNUS_USER=cygnus
CONFIG_FOLDER=/usr/cygnus/conf
CONFIG_FILE=/usr/cygnus/conf/agent.conf
AGENT_NAME=cygnusagent
LOGFILE_NAME=cygnus.log
ADMIN_PORT=8081
POLLING_INTERVAL=30

But to no avail Cygnus is still not persisting data in PostGres.但无济于事 Cygnus 仍然没有在 PostGres 中持久化数据。

Here are some logs from the Cygnus service:以下是 Cygnus 服务的一些日志：

time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation start false:ACTION:Starts the instance

time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: start()

time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: stop

time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation stop false:ACTION:Stops the instance

time=2022-03-10T17:02:12.430Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: stop()

time=2022-03-10T17:02:12.431Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.util.component.Container

time=2022-03-10T17:02:12.431Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[291] : Annotations not found for: org.eclipse.jetty.util.component.Container

time=2022-03-10T17:02:12.431Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.util.component.Destroyable

time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: destroy

time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation destroy false:ACTION:Destroys this component

time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: destroy()

time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.util.component.Dumpable

time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: dump

time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation dump false:INFO:Dump the nested Object state as a String

time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: dump()

time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.server.Handler

time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[305] : Attribute Annotation found for: getServer

time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineAttribute | msg=org.eclipse.jetty.jmx.ObjectMBean[657] : defineAttribute server false:true:class org.eclipse.jetty.server.handler.ErrorHandler:the jetty server for this handler

time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineAttribute | msg=org.eclipse.jetty.jmx.ObjectMBean[729] : passed convert checks server for type class org.eclipse.jetty.server.Server

time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: destroy

time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation destroy false:ACTION:destroy associated resources

time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: destroy()

time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[208] : Registered org.eclipse.jetty.server.handler:type=errorhandler,id=0

time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=addBean | msg=org.eclipse.jetty.util.component.ContainerLifeCycle[322] : org.eclipse.jetty.server.Server@52e18a5d added {org.eclipse.jetty.server.handler.ErrorHandler@4a257334,AUTO}

time=2022-03-10T17:02:12.436Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=doStart | msg=org.eclipse.jetty.server.Server[372] : jetty-9.4.6.v20170531

time=2022-03-10T17:02:12.471Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=doStart | msg=org.eclipse.jetty.server.handler.AbstractHandler[110] : starting org.eclipse.jetty.server.Server@52e18a5d

time=2022-03-10T17:02:12.471Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=setStarting | msg=org.eclipse.jetty.util.component.AbstractLifeCycle[185] : starting qtp260686092{STOPPED,8<=0<=200,i=0,q=0}

time=2022-03-10T17:02:12.488Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=setStarted | msg=org.eclipse.jetty.util.component.AbstractLifeCycle[177] : STARTED @1893ms qtp260686092{STARTED,8<=8<=200,i=7,q=0}

time=2022-03-10T17:02:12.488Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[131] : beanAdded ServerConnector@cc1c486{HTTP/1.1,[http/1.1]}{0.0.0.0:5055}->org.eclipse.jetty.server.Server@52e18a5d

time=2022-03-10T17:02:12.489Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[131] : beanAdded ServerConnector@cc1c486{HTTP/1.1,[http/1.1]}{0.0.0.0:5055}->qtp260686092{STARTED,8<=8<=200,i=8,q=0}

time=2022-03-10T17:02:12.489Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[131] : beanAdded ServerConnector@cc1c486{HTTP/1.1,[http/1.1]}{0.0.0.0:5055}->org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@36176a5b

ing | msg=org.eclipse.jetty.util.component.AbstractLifeCycle[185] : starting o.e.j.s.ServletContextHandler@30833221{/,null,UNAVAILABLE}

time=2022-03-10T17:02:12.551Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=<clinit> | msg=org.eclipse.jetty.http.PreEncodedHttpField[64] : HttpField encoders loaded: [org.eclipse.jetty.http.Http1FieldPreEncoder@5b3a334b]

Thanks in advance.提前致谢。

1 个解决方案

解决方案1
 0  2022-03-11 14:18:27

It's all good now I found the issue.现在一切都很好，我发现了问题。

My subscription for Orion uses port 5050 (which is the port exposed if the agent is MySQL) for PostGresSINK the port exposed by the agent is 5055 so I updated the subscription and it works fine now.我对 Orion 的订阅使用端口 5050（如果代理是 MySQL，则该端口公开）用于 PostGresSINK，代理公开的端口是 5055，所以我更新了订阅，它现在工作正常。

"url": "http://cygnus:5055/notify" “网址”：“http://cygnus:5055/notify”

Always be careful with copy N paste I just learned it the hard way ^^总是小心复制 N 粘贴我刚刚学到它 ^^

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 Docker卷不保留数据 - Docker volume does not persist data Cygnus-NGSI不会在PostgreSQL中保存数据 - Cygnus-NGSI won't save data in PostgreSQL 设置Cygnus-PostgreSQL以进行历史数据持久化 - Setting up Cygnus-PostgreSQL for historical data persistance 当实体 id 包含特殊字符时，Fiware Cygnus PostgreSQL sink 不会创建表 - Fiware Cygnus PostgreSQL sink does not create table when entity id contains special characters 更新 postSQL 和 mongodb 之间的查询性能 - update query performance between postSQL and mongodb 连接系统，其中包括 OPC UA IoT Agent、Orion Context Broker。 Cygnus 和 Postgres 的历史数据 - Connect system which includes OPC UA IoT Agent, Orion Context Broker. Cygnus and Historic data with Postgres 使用R中的ssh隧道连接到数据库PostSQL - Connect to database PostSQL using ssh tunnel in R 使用逗号分隔符填充表时出现 postsql 问题 - postsql trouble when populating a table with comma separator 无法使用 kotlin 和 spring 数据持久化数据 - Cannot persist data using kotlin and spring data 在Postgresql中运行Cygnus的问题 - Problems running Cygnus with Postgresql
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM