[Fiware-general-help] [FIWARE-JIRA] (HELP-12977) [Fiware-tech-help] IMPACT: Delivering CartoDb events through Cygnus

FW External User (JIRA) jira-help-desk at jira.fiware.org
Tue Jan 9 15:32:00 CET 2018 

     [ https://jira.fiware.org/browse/HELP-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

FW External User updated HELP-12977:
------------------------------------

Comment by peter at familiohq.com : 

Thanks! I'll run the scripts and get back to you.


On Mon, Jan 8, 2018 at 4:22 PM Help-Desk <jira-help-desk at jira.fiware.org>
wrote:

>
>



> [Fiware-tech-help] IMPACT: Delivering CartoDb events through Cygnus
> -------------------------------------------------------------------
>
>                 Key: HELP-12977
>                 URL: https://jira.fiware.org/browse/HELP-12977
>             Project: Help-Desk
>          Issue Type: extRequest
>          Components: FIWARE-TECH-HELP
>            Reporter: FW External User
>            Assignee: Andres Muñoz
>         Attachments: carto-cygnus-log.txt, cygnus-log.txt, notification-json-geom.sh
>
>
> I've setup Orion and Cygnus so whenever a new entity is created in Orion
> it's transferring its data to Carto using the Cygnus CartoDbSink.
> However I'm having some issues with this implementation. I'm getting the
> data correctly from Orion to Cygnus but when the sink tries to forward
> those data to Carto I'm getting this error message (also attached for
> better formatting) when trying to persist it in Carto
> cygnus_1             | time=2017-11-23T13:40:11.175Z | lvl=INFO |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=getEvents |
> msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[294] :
> [NGSIRestHandler] Starting internal transaction
> (efab5d17-12c3-4716-8b1b-15352ec6ae18)
> cygnus_1             | time=2017-11-23T13:40:11.176Z | lvl=INFO |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=getEvents |
> msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[312] :
> [NGSIRestHandler] Received data
> ({"subscriptionId":"5a16cd57fde41d458740a0b4","originator":"localhost","contextResponses":[{"contextElement":{"type":"Event","isPattern":"false","id":"getif","attributes":[{"name":"location","type":"geo:point","value":"79.00555,
> -148.99808"}]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]})
> cygnus_1             | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=getEvents |
> msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[320] :
> [NGSIRestHandler] Parsed NotifyContextRequest:
> {"subscriptionId":"5a16cd57fde41d458740a0b4","originator":"localhost","contextResponses":[{"contextElement":{"id":"getif","type":"Event","isPattern":"false","attributes":[{"name":"location","type":"geo:point","value":"79.00555,
> -148.99808","metadatas":[]}]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]}
> cygnus_1             | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=getEvents |
> msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[345] :
> [NGSIRestHandler] NGSI event created for ContextElementResponse:
> {"contextElement":{"id":"getif","type":"Event","isPattern":"false","attributes":[{"name":"location","type":"geo:point","value":"79.00555,
> -148.99808","metadatas":[]}]},"statusCode":{"code":"200","reasonPhrase":"OK"}}
> cygnus_1             | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=getEvents |
> msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[350] :
> [NGSIRestHandler] Header added to NGSI event (fiware-service: default)
> cygnus_1             | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=getEvents |
> msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[353] :
> [NGSIRestHandler] Header added to NGSI event (fiware-servicepath: /)
> cygnus_1             | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=getEvents |
> msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[356] :
> [NGSIRestHandler] Header added to NGSI event (fiware-correlator:
> d3be117c-d053-11e7-ad11-0242ac120009)
> cygnus_1             | time=2017-11-23T13:40:11.240Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=getEvents |
> msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[359] :
> [NGSIRestHandler] Header added to NGSI event (transaction-id:
> efab5d17-12c3-4716-8b1b-15352ec6ae18)
> cygnus_1             | time=2017-11-23T13:40:11.240Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=getEvents |
> msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[385] :
> [NGSIRestHandler] NGSI events put in the channel, ids=1516398619
> cygnus_1             | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=intercept |
> msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[84] :
> [gi] Event intercepted, id=1516398619
> cygnus_1             | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=intercept |
> msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[117] :
> [gi] Adding flume event header (notified-entity: getif_Event)
> cygnus_1             | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=intercept |
> msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[120] :
> [gi] Adding flume event header (grouped-entity: getif_Event)
> cygnus_1             | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=intercept |
> msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[123] :
> [gi] Adding flume event header (grouped-servicepath: /)
> cygnus_1             | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=intercept |
> msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[127] :
> [gi] Event put in the channel, id=1516398619
> orion_1              | time=Thursday 23 Nov 13:40:11 2017.243Z | lvl=INFO |
> corr=N/A | trans=1511444367-204-00000000002 | from=pending | srv=pending |
> subsrv=pending | comp=Orion |
> op=httpRequestSend.cpp[620]:httpRequestSendWithCurl | msg=Notification
> Successfully Sent to http://staging.familiohq.com:5050/notify
> orion_1              | time=Thursday 23 Nov 13:40:11 2017.243Z | lvl=INFO |
> corr=N/A | trans=1511444367-204-00000000002 | from=pending | srv=pending |
> subsrv=pending | comp=Orion | op=logMsg.h[1916]:lmTransactionEnd |
> msg=Transaction ended
> cygnus_1             | time=2017-11-23T13:40:11.244Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=processNewBatches |
> msg=com.telefonica.iot.cygnus.sinks.NGSISink[554] : Batch completed
> cygnus_1             | time=2017-11-23T13:40:11.244Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=persistBatch |
> msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[381] : [cartodb-sink]
> Processing sub-batch regarding the default_/_getif_Event destination
> cygnus_1             | time=2017-11-23T13:40:11.254Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=aggregate |
> msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink$CartoDBAggregator[570]
> : [cartodb-sink] Processing context element (id=getif, type=Event)
> cygnus_1             | time=2017-11-23T13:40:11.254Z | lvl=DEBUG |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=aggregate |
> msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink$CartoDBAggregator[592]
> : [cartodb-sink] Processing context attribute (name=location,
> type=geo:point)
> cygnus_1             | time=2017-11-23T13:40:11.254Z | lvl=INFO |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=persistRawAggregation |
> msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[616] : [cartodb-sink]
> Persisting data at NGSICartoDBSink. Schema (default), Table
> (x002fxffffgetifxffffx0045vent), Data
> (('2017-11-23T13:40:11.241Z','/','getif','Event',ST_SetSRID(ST_MakePoint(79.00555,-148.99808),
> 4326)))
> cygnus_1             | time=2017-11-23T13:40:11.255Z | lvl=ERROR |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-*15352ec6ae18
> | srv=default | subsrv=/ | comp=cygnus-ngsi | op=run |
> msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
> event. Exception follows.*
> *cygnus_1             | java.lang.NullPointerException*
> *cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSICartoDBSink.persistRawAggregation(NGSICartoDBSink.java:625)*
> *cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSICartoDBSink.persistBatch(NGSICartoDBSink.java:417)*
> *cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:558)*
> *cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)*
> *cygnus_1             | at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)*
> *cygnus_1             | at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
> *cygnus_1             | at java.lang.Thread.run(Thread.java:748)*
> cygnus_1             | time=2017-11-23T13:40:16.256Z | lvl=ERROR |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=run |
> msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
> event. Exception follows.
> cygnus_1             | java.lang.IllegalStateException: begin() called when
> transaction is OPEN!
> cygnus_1             | at
> com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> cygnus_1             | at
> org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)
> cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)
> cygnus_1             | at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> cygnus_1             | at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> cygnus_1             | at java.lang.Thread.run(Thread.java:748)
> cygnus_1             | time=2017-11-23T13:40:18.648Z | lvl=DEBUG | corr=N/A
> | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=run |
> msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[126]
> : Checking file:/opt/apache-flume/conf/agent.conf for changes
> cygnus_1             | time=2017-11-23T13:40:18.654Z | lvl=WARN | corr=N/A
> | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=<init> |
> msg=com.telefonica.iot.cygnus.interceptors.CygnusGroupingRules[49] : No
> grouping rules have been read. Details:
> /usr/cygnus/conf/grouping_rules.conf (No such file or directory)
> cygnus_1             | time=2017-11-23T13:40:21.257Z | lvl=ERROR |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=run |
> msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
> event. Exception follows.
> cygnus_1             | java.lang.IllegalStateException: begin() called when
> transaction is OPEN!
> cygnus_1             | at
> com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> cygnus_1             | at
> org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)
> cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)
> cygnus_1             | at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> cygnus_1             | at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> cygnus_1             | at java.lang.Thread.run(Thread.java:748)
> cygnus_1             | time=2017-11-23T13:40:26.258Z | lvl=ERROR |
> corr=d3be117c-d053-11e7-ad11-0242ac120009 |
> trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
> comp=cygnus-ngsi | op=run |
> msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
> event. Exception follows.
> cygnus_1             | java.lang.IllegalStateException: begin() called when
> transaction is OPEN!
> cygnus_1             | at
> com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> cygnus_1             | at
> org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)
> cygnus_1             | at
> com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)
> cygnus_1             | at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> cygnus_1             | at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> cygnus_1             | at java.lang.Thread.run(Thread.java:748)
> Are you able to help me figure out why I can't send this information to
> carto?
> -- 
> *Peter Theill*
> CTO
> +45 6171 5096 <61%2071%2050%2096>
> peter at familiohq.com
> *Familio*
> Artillerivej 86
> DK-2300 Copenhagen S
> https://familiohq.com/
> __________________________________________________________________________________________
> You can get more information about our cookies and privacy policies on the following links:
> - http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE_Privacy_Policy
> - http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Cookies_Policy_FIWARE
> Fiware-tech-help mailing list
> Fiware-tech-help at lists.fiware.org
> https://lists.fiware.org/listinfo/fiware-tech-help
> [Created via e-mail received from: Peter Theill <peter at familiohq.com>]



--
This message was sent by Atlassian JIRA
(v6.4.1#64016)

More information about the Fiware-general-help mailing list

You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy   Cookies policy