[Backlogmanager] [FIWARE-JIRA] (HELP-19318) [fiware-stackoverflow] Cygnus does not persist data into PostSQL

[Fernando Lopez (JIRA)]

     [ https://jira.fiware.org/browse/HELP-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fernando Lopez reassigned HELP-19318:
-------------------------------------

    Assignee: Backlog Manager

> [fiware-stackoverflow] Cygnus does not persist data into PostSQL
> ----------------------------------------------------------------
>
>                 Key: HELP-19318
>                 URL: https://jira.fiware.org/browse/HELP-19318
>             Project: Help-Desk
>          Issue Type: Monitor
>          Components: FIWARE-TECH-HELP
>            Reporter: Backlog Manager
>            Assignee: Backlog Manager
>              Labels: fiware, fiware-cygnus, postgresql
>
> Created question in FIWARE Q/A platform on 10-03-2022 at 17:03
> {color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/71428255/cygnus-does-not-persist-data-into-postsql
> +Question:+
> Cygnus does not persist data into PostSQL
> +Description:+
> I am using the following tutorial to persist data in a PostGresQL database via Cygnus.
> https://documenter.getpostman.com/view/513743/RWEcR2DC
> When using a MySQL database with Cygnus everything works fine as you can see below:
> mysql> SHOW tables FROM openiot;
> +-------------------------------------------------------+
> | Tables_in_openiot                                     |
> +-------------------------------------------------------+
> | urn_ngsi-ld_TemperatureSensor_11401_TemperatureSensor |
> | urn_ngsi-ld_TemperatureSensor_12435_TemperatureSensor |
> | urn_ngsi-ld_TemperatureSensor_13477_TemperatureSensor |
> | urn_ngsi-ld_TemperatureSensor_13828_TemperatureSensor |
> | urn_ngsi-ld_TemperatureSensor_15205_TemperatureSensor |
> | urn_ngsi-ld_TemperatureSensor_1546_TemperatureSensor  |
> +-------------------------------------------------------+
> 109 rows in set (0.00 sec)
> But when using PostGres as backend, the NGSIPostgreSQLSink does not create tables.
> Here is my setup on docker:
>   postgres:
>     image: postgres:latest
>     hostname: postgres
>     expose:
>       - "5432"
>     ports:
>       - "5432:5432"
>     environment:
>       - "POSTGRES_PASSWORD=password"
>       - "POSTGRES_USER=postgres"
>       - "POSTGRES_DB=postgres"
>     volumes:
>       - data-postgres:/var/lib/postgresql/data
>     
>   cygnus:
>     image: fiware/cygnus-ngsi:latest
>     hostname: cygnus
>     depends_on:
>         - postgres
>     expose:
>         - "5080"
>     ports:
>         - "5055:5055"
>         - "5080:5080"
>     environment:
>         - "CYGNUS_POSTGRESQL_HOST=postgres"
>         - "CYGNUS_POSTGRESQL_PORT=5432"
>         - "CYGNUS_POSTGRESQL_USER=postgres"
>         - "CYGNUS_POSTGRESQL_PASS=password"
>         - "CYGNUS_POSTGRESQL_ENABLE_CACHE=true"
>         - "CYGNUS_POSTGRESQL_SERVICE_PORT=5055"
>         - "CYGNUS_LOG_LEVEL=DEBUG"
>         - "CYGNUS_API_PORT=5080"
>         - "CYGNUS_SERVICE_PORT=5055"
> If I list the schemas with a PostGres client there is no openiot schema:
> postgres=# \dn
>   List of schemas
>   Name  |  Owner
> --------+----------
>  public | postgres
> (1 row)
> I also tried to mount the following agent.conf and cygnus_instance.conf files in docker:
> docker
> volumes:
>    - ./conf/cygnus:/opt/apache-flume/conf/agent:ro
> agent.conf
> cygnus-ngsi.sources = http-source
> cygnus-ngsi.sinks = postgresql-sink
> cygnus-ngsi.channels = postgresql-channel
> cygnus-ngsi.sources.http-source.channels = postgresql-channel
> cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
> cygnus-ngsi.sources.http-source.port = 5050
> cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
> cygnus-ngsi.sources.http-source.handler.notification_target = /notify
> cygnus-ngsi.sources.http-source.handler.default_service = default
> cygnus-ngsi.sources.http-source.handler.default_service_path = /
> cygnus-ngsi.sources.http-source.interceptors = ts gi
> cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
> cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
> cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
> cygnus-ngsi.sources.http-source.interceptors.nmi.type = com.telefonica.iot.cygnus.interceptors.NGSINameMappingsInterceptor$Builder
> cygnus-ngsi.sources.http-source.interceptors.nmi.name_mappings_conf_file = /usr/cygnus/conf/name_mappings.conf
> cygnus-ngsi.sinks.postgresql-sink.channel = postgresql-channel
> cygnus-ngsi.sinks.postgresql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink
> cygnus-ngsi.sinks.postgresql-sink.enable_encoding = false
> cygnus-ngsi.sinks.postgresql-sink.enable_name_mappings = false
> cygnus-ngsi.sinks.postgresql-sink.enable_grouping = false
> cygnus-ngsi.sinks.postgresql-sink.enable_lowercase = false
> cygnus-ngsi.sinks.postgresql-sink.postgresql_host = postgres
> cygnus-ngsi.sinks.postgresql-sink.postgresql_port = 5432
> cygnus-ngsi.sinks.postgresql-sink.postgresql_database = postgres
> cygnus-ngsi.sinks.postgresql-sink.postgresql_username = postgres
> cygnus-ngsi.sinks.postgresql-sink.postgresql_password = password
> cygnus-ngsi.sinks.postgresql-sink.attr_persistence = column
> cygnus-ngsi.sinks.postgresql-sink.data_model = by-service-path
> cygnus-ngsi.sinks.postgresql-sink.batch_size = 100
> cygnus-ngsi.sinks.postgresql-sink.batch_timeout = 30
> cygnus-ngsi.sinks.postgresql-sink.batch_ttl = 10
> cygnus-ngsi.sinks.postgresql-sink.persist_errors = true
> cygnus-ngsi.channels.postgresql-channel.type = memory
> cygnus-ngsi.channels.postgresql-channel.capacity = 100000
> cygnus-ngsi.channels.postgresql-channel.transactionCapacity = 10000
> cygnus_instance.conf
> CYGNUS_USER=cygnus
> CONFIG_FOLDER=/usr/cygnus/conf
> CONFIG_FILE=/usr/cygnus/conf/agent.conf
> AGENT_NAME=cygnusagent
> LOGFILE_NAME=cygnus.log
> ADMIN_PORT=8081
> POLLING_INTERVAL=30
> But to no avail Cygnus is still not persisting data in PostGres.
> Here are some logs from the Cygnus service:
> time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation start false:ACTION:Starts the instance
> time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: start()
> time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: stop
> time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation stop false:ACTION:Stops the instance
> time=2022-03-10T17:02:12.430Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: stop()
> time=2022-03-10T17:02:12.431Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.util.component.Container
> time=2022-03-10T17:02:12.431Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[291] : Annotations not found for: org.eclipse.jetty.util.component.Container
> time=2022-03-10T17:02:12.431Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.util.component.Destroyable
> time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: destroy
> time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation destroy false:ACTION:Destroys this component
> time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: destroy()
> time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.util.component.Dumpable
> time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: dump
> time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation dump false:INFO:Dump the nested Object state as a String
> time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: dump()
> time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.server.Handler
> time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[305] : Attribute Annotation found for: getServer
> time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineAttribute | msg=org.eclipse.jetty.jmx.ObjectMBean[657] : defineAttribute server false:true:class org.eclipse.jetty.server.handler.ErrorHandler:the jetty server for this handler
> time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineAttribute | msg=org.eclipse.jetty.jmx.ObjectMBean[729] : passed convert checks server for type class org.eclipse.jetty.server.Server
> time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: destroy
> time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation destroy false:ACTION:destroy associated resources
> time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: destroy()
> time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[208] : Registered org.eclipse.jetty.server.handler:type=errorhandler,id=0
> time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=addBean | msg=org.eclipse.jetty.util.component.ContainerLifeCycle[322] : org.eclipse.jetty.server.Server at 52e18a5d added {org.eclipse.jetty.server.handler.ErrorHandler at 4a257334,AUTO}
> time=2022-03-10T17:02:12.436Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=doStart | msg=org.eclipse.jetty.server.Server[372] : jetty-9.4.6.v20170531
> time=2022-03-10T17:02:12.471Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=doStart | msg=org.eclipse.jetty.server.handler.AbstractHandler[110] : starting org.eclipse.jetty.server.Server at 52e18a5d
> time=2022-03-10T17:02:12.471Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=setStarting | msg=org.eclipse.jetty.util.component.AbstractLifeCycle[185] : starting qtp260686092{STOPPED,8<=0<=200,i=0,q=0}
> time=2022-03-10T17:02:12.488Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=setStarted | msg=org.eclipse.jetty.util.component.AbstractLifeCycle[177] : STARTED @1893ms qtp260686092{STARTED,8<=8<=200,i=7,q=0}
> time=2022-03-10T17:02:12.488Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[131] : beanAdded ServerConnector at cc1c486{HTTP/1.1,[http/1.1]}{0.0.0.0:5055}->org.eclipse.jetty.server.Server at 52e18a5d
> time=2022-03-10T17:02:12.489Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[131] : beanAdded ServerConnector at cc1c486{HTTP/1.1,[http/1.1]}{0.0.0.0:5055}->qtp260686092{STARTED,8<=8<=200,i=8,q=0}
> time=2022-03-10T17:02:12.489Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[131] : beanAdded ServerConnector at cc1c486{HTTP/1.1,[http/1.1]}{0.0.0.0:5055}->org.eclipse.jetty.util.thread.ScheduledExecutorScheduler at 36176a5b
> ing | msg=org.eclipse.jetty.util.component.AbstractLifeCycle[185] : starting o.e.j.s.ServletContextHandler at 30833221{/,null,UNAVAILABLE}
> time=2022-03-10T17:02:12.551Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=<clinit> | msg=org.eclipse.jetty.http.PreEncodedHttpField[64] : HttpField encoders loaded: [org.eclipse.jetty.http.Http1FieldPreEncoder at 5b3a334b]
> Thanks in advance.



--
This message was sent by Atlassian JIRA
(v6.4.1#64016)