[ https://jira.fiware.org/browse/HELP-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fernando Lopez updated HELP-8364:
---------------------------------
HD-Chapter: Data
Description:
Created question in FIWARE Q/A platform on 08-03-2017 at 14:03
{color: red}Please, ANSWER this question AT{color} http://stackoverflow.com/questions/42672058/fiware-cygnus-error-collection-already-exists-when-cygnus-receives-notificati
+Question:+
Fiware Cygnus - Error: collection already exists when Cygnus receives notifications two or more times
+Description:+
I am using Fiware Cygnus configured with STH. When Cygnus receives two or more notifications of changes in any entity it can't send the new value to STH. It logs the error collection already exists and the change is not saved. In the first notification everything goes well. The full message is:
* I am using cygnus in a container docker as described in the documentation.
* I used mongodb-ip:27017 instead real ip to post here.
time=2017-03-08T11:51:05.164Z | lvl=ERROR |
corr=53d86140-03f5-11e7-a70e-080027f6529d |
trans=236416c2-776e-4cc0-91dc-29bca203ea2a | srv=red | subsrv=/red/red |
comp=cygnus-ngsi | op=processRollbackedBatches |msg=com.telefonica.iot.cygnus.sinks.NGSISink[394] :
Persistence error. Message: -, Command failed with error -1:
'collection already exists' on server <mongodb-ip>:27017. The full response is
{ "ok" : 0.0, "errmsg" : "collection already exists" }, Stack trace:
[com.telefonica.iot.cygnus.sinks.NGSISTHSink.persistOne(NGSISTHSink.java:158),
com.telefonica.iot.cygnus.sinks.NGSISTHSink.persistBatch(NGSISTHSink.java:93),
com.telefonica.iot.cygnus.sinks.NGSISink.processRollbackedBatches(NGSISink.java:387),
com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370),
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68),
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147),
java.lang.Thread.run(Thread.java:745)]
time=2017-03-08T11:51:05.164Z | lvl=INFO | corr=53d86140-03f5-11e7-a70e-080027f6529d | trans=236416c2-776e-4cc0-91dc-29bca203ea2a | srv=red | subsrv=/red/red | comp=cygnus-ngsi | op=doRollbackAgain | msg=com.telefonica.iot.cygnus.sinks.NGSISink[464] : Finishing internal transaction (53d86140-03f5-11e7-a70e-080027f6529d), this was retry #10
The configuration that I am using in my agent.conf (Cygnus) is:
cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = sth-sink
cygnus-ngsi.channels = sth-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = sth-channel
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf
cygnus-ngsi.sinks.sth-sink.type = com.telefonica.iot.cygnus.sinks.NGSISTHSink
cygnus-ngsi.sinks.sth-sink.channel = sth-channel
#cygnus-ngsi.sinks.sth-sink.enable_encoding = false
#cygnus-ngsi.sinks.sth-sink.enable_grouping = false
#cygnus-ngsi.sinks.sth-sink.enable_name_mappings = false
#cygnus-ngsi.sinks.sth-sink.enable_lowercase = false
cygnus-ngsi.sinks.sth-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.sth-sink.mongo_hosts = <mongodb-ip>:27017
cygnus-ngsi.sinks.sth-sink.mongo_username =
cygnus-ngsi.sinks.sth-sink.mongo_password =
cygnus-ngsi.sinks.sth-sink.db_prefix = sth_
cygnus-ngsi.sinks.sth-sink.collection_prefix = sth_
cygnus-ngsi.sinks.sth-sink.resolutions = day,hour,minute
#cygnus-ngsi.sinks.sth-sink.batch_size = 1
#cygnus-ngsi.sinks.sth-sink.batch_timeout = 30
#cygnus-ngsi.sinks.sth-sink.batch_ttl = 10
#cygnus-ngsi.sinks.sth-sink.data_expiration = 0
#cygnus-ngsi.sinks.sth-sink.ignore_white_spaces = true
cygnus-ngsi.channels.sth-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
cygnus-ngsi.channels.sth-channel.capacity = 1000
cygnus-ngsi.channels.sth-channel.transactionCapacity = 100
In the STH side I have the following conf:
var config = {};
// STH server configuration
//--------------------------
config.server = {
host: '10.0.2.15',
port: '8666',
defaultService: 'testservice',
defaultServicePath: '/testservicepath',
filterOutEmpty: 'true',
aggregationBy: ['day', 'hour', 'minute'],
temporalDir: 'temp'
};
// Database configuration
//------------------------
config.database = {
dataModel: 'collection-per-entity',
user: '',
password: '',
URI: 'localhost:27017',
replicaSet: '',
prefix: 'sth_',
collectionPrefix: 'sth_',
poolSize: '5',
shouldStore: 'both',
truncation: {
expireAfterSeconds: '0',
size: '0',
max: '0'
},
ignoreBlankSpaces: 'true',
nameMapping: {
enabled: 'false',
configFile: './name-mapping.json'
},
nameEncoding: 'false'
};
// Logging configuration
//------------------------
config.logging = {
level: 'info',
NODE_ENV variable is set to 'development'.
format: 'pipe',
proofOfLifeInterval: '60'
};
module.exports = config;
was:
Created question in FIWARE Q/A platform on 08-03-2017 at 14:03
{color: red}Please, ANSWER this question AT{color} http://stackoverflow.com/questions/42672058/fiware-cygnus-error-collection-already-exists-when-cygnus-receives-notificati
+Question:+
Fiware Cygnus - Error: collection already exists when Cygnus receives notifications two or more times
+Description:+
I am using Fiware Cygnus configured with STH. When Cygnus receives two or more notifications of changes in any entity it can't send the new value to STH. It logs the error collection already exists and the change is not saved. In the first notification everything goes well. The full message is:
* I am using cygnus in a container docker as described in the documentation.
* I used mongodb-ip:27017 instead real ip to post here.
time=2017-03-08T11:51:05.164Z | lvl=ERROR |
corr=53d86140-03f5-11e7-a70e-080027f6529d |
trans=236416c2-776e-4cc0-91dc-29bca203ea2a | srv=red | subsrv=/red/red |
comp=cygnus-ngsi | op=processRollbackedBatches |msg=com.telefonica.iot.cygnus.sinks.NGSISink[394] :
Persistence error. Message: -, Command failed with error -1:
'collection already exists' on server <mongodb-ip>:27017. The full response is
{ "ok" : 0.0, "errmsg" : "collection already exists" }, Stack trace:
[com.telefonica.iot.cygnus.sinks.NGSISTHSink.persistOne(NGSISTHSink.java:158),
com.telefonica.iot.cygnus.sinks.NGSISTHSink.persistBatch(NGSISTHSink.java:93),
com.telefonica.iot.cygnus.sinks.NGSISink.processRollbackedBatches(NGSISink.java:387),
com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370),
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68),
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147),
java.lang.Thread.run(Thread.java:745)]
time=2017-03-08T11:51:05.164Z | lvl=INFO | corr=53d86140-03f5-11e7-a70e-080027f6529d | trans=236416c2-776e-4cc0-91dc-29bca203ea2a | srv=red | subsrv=/red/red | comp=cygnus-ngsi | op=doRollbackAgain | msg=com.telefonica.iot.cygnus.sinks.NGSISink[464] : Finishing internal transaction (53d86140-03f5-11e7-a70e-080027f6529d), this was retry #10
The configuration that I am using in my agent.conf (Cygnus) is:
cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = sth-sink
cygnus-ngsi.channels = sth-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = sth-channel
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf
cygnus-ngsi.sinks.sth-sink.type = com.telefonica.iot.cygnus.sinks.NGSISTHSink
cygnus-ngsi.sinks.sth-sink.channel = sth-channel
#cygnus-ngsi.sinks.sth-sink.enable_encoding = false
#cygnus-ngsi.sinks.sth-sink.enable_grouping = false
#cygnus-ngsi.sinks.sth-sink.enable_name_mappings = false
#cygnus-ngsi.sinks.sth-sink.enable_lowercase = false
cygnus-ngsi.sinks.sth-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.sth-sink.mongo_hosts = <mongodb-ip>:27017
cygnus-ngsi.sinks.sth-sink.mongo_username =
cygnus-ngsi.sinks.sth-sink.mongo_password =
cygnus-ngsi.sinks.sth-sink.db_prefix = sth_
cygnus-ngsi.sinks.sth-sink.collection_prefix = sth_
cygnus-ngsi.sinks.sth-sink.resolutions = day,hour,minute
#cygnus-ngsi.sinks.sth-sink.batch_size = 1
#cygnus-ngsi.sinks.sth-sink.batch_timeout = 30
#cygnus-ngsi.sinks.sth-sink.batch_ttl = 10
#cygnus-ngsi.sinks.sth-sink.data_expiration = 0
#cygnus-ngsi.sinks.sth-sink.ignore_white_spaces = true
cygnus-ngsi.channels.sth-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
cygnus-ngsi.channels.sth-channel.capacity = 1000
cygnus-ngsi.channels.sth-channel.transactionCapacity = 100
In the STH side I have the following conf:
var config = {};
// STH server configuration
//--------------------------
config.server = {
host: '10.0.2.15',
port: '8666',
defaultService: 'testservice',
defaultServicePath: '/testservicepath',
filterOutEmpty: 'true',
aggregationBy: ['day', 'hour', 'minute'],
temporalDir: 'temp'
};
// Database configuration
//------------------------
config.database = {
dataModel: 'collection-per-entity',
user: '',
password: '',
URI: 'localhost:27017',
replicaSet: '',
prefix: 'sth_',
collectionPrefix: 'sth_',
poolSize: '5',
shouldStore: 'both',
truncation: {
expireAfterSeconds: '0',
size: '0',
max: '0'
},
ignoreBlankSpaces: 'true',
nameMapping: {
enabled: 'false',
configFile: './name-mapping.json'
},
nameEncoding: 'false'
};
// Logging configuration
//------------------------
config.logging = {
level: 'info',
NODE_ENV variable is set to 'development'.
format: 'pipe',
proofOfLifeInterval: '60'
};
module.exports = config;
HD-Enabler: Cosmos
> [fiware-stackoverflow] Fiware Cygnus - Error: collection already exists when Cygnus receives notifications two or more times
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HELP-8364
> URL: https://jira.fiware.org/browse/HELP-8364
> Project: Help-Desk
> Issue Type: Monitor
> Components: FIWARE-TECH-HELP
> Reporter: Backlog Manager
> Assignee: Francisco Romero
> Labels: fiware, fiware-cygnus, fiware-sth-comet
>
> Created question in FIWARE Q/A platform on 08-03-2017 at 14:03
> {color: red}Please, ANSWER this question AT{color} http://stackoverflow.com/questions/42672058/fiware-cygnus-error-collection-already-exists-when-cygnus-receives-notificati
> +Question:+
> Fiware Cygnus - Error: collection already exists when Cygnus receives notifications two or more times
> +Description:+
> I am using Fiware Cygnus configured with STH. When Cygnus receives two or more notifications of changes in any entity it can't send the new value to STH. It logs the error collection already exists and the change is not saved. In the first notification everything goes well. The full message is:
> * I am using cygnus in a container docker as described in the documentation.
> * I used mongodb-ip:27017 instead real ip to post here.
> time=2017-03-08T11:51:05.164Z | lvl=ERROR |
> corr=53d86140-03f5-11e7-a70e-080027f6529d |
> trans=236416c2-776e-4cc0-91dc-29bca203ea2a | srv=red | subsrv=/red/red |
> comp=cygnus-ngsi | op=processRollbackedBatches |msg=com.telefonica.iot.cygnus.sinks.NGSISink[394] :
> Persistence error. Message: -, Command failed with error -1:
> 'collection already exists' on server <mongodb-ip>:27017. The full response is
> { "ok" : 0.0, "errmsg" : "collection already exists" }, Stack trace:
> [com.telefonica.iot.cygnus.sinks.NGSISTHSink.persistOne(NGSISTHSink.java:158),
> com.telefonica.iot.cygnus.sinks.NGSISTHSink.persistBatch(NGSISTHSink.java:93),
> com.telefonica.iot.cygnus.sinks.NGSISink.processRollbackedBatches(NGSISink.java:387),
> com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370),
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68),
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147),
> java.lang.Thread.run(Thread.java:745)]
> time=2017-03-08T11:51:05.164Z | lvl=INFO | corr=53d86140-03f5-11e7-a70e-080027f6529d | trans=236416c2-776e-4cc0-91dc-29bca203ea2a | srv=red | subsrv=/red/red | comp=cygnus-ngsi | op=doRollbackAgain | msg=com.telefonica.iot.cygnus.sinks.NGSISink[464] : Finishing internal transaction (53d86140-03f5-11e7-a70e-080027f6529d), this was retry #10
> The configuration that I am using in my agent.conf (Cygnus) is:
> cygnus-ngsi.sources = http-source
> cygnus-ngsi.sinks = sth-sink
> cygnus-ngsi.channels = sth-channel
> cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
> cygnus-ngsi.sources.http-source.channels = sth-channel
> cygnus-ngsi.sources.http-source.port = 5050
> cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
> cygnus-ngsi.sources.http-source.handler.notification_target = /notify
> cygnus-ngsi.sources.http-source.handler.default_service = default
> cygnus-ngsi.sources.http-source.handler.default_service_path = /
> cygnus-ngsi.sources.http-source.interceptors = ts gi
> cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
> cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
> cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf
> cygnus-ngsi.sinks.sth-sink.type = com.telefonica.iot.cygnus.sinks.NGSISTHSink
> cygnus-ngsi.sinks.sth-sink.channel = sth-channel
> #cygnus-ngsi.sinks.sth-sink.enable_encoding = false
> #cygnus-ngsi.sinks.sth-sink.enable_grouping = false
> #cygnus-ngsi.sinks.sth-sink.enable_name_mappings = false
> #cygnus-ngsi.sinks.sth-sink.enable_lowercase = false
> cygnus-ngsi.sinks.sth-sink.data_model = dm-by-entity
> cygnus-ngsi.sinks.sth-sink.mongo_hosts = <mongodb-ip>:27017
> cygnus-ngsi.sinks.sth-sink.mongo_username =
> cygnus-ngsi.sinks.sth-sink.mongo_password =
> cygnus-ngsi.sinks.sth-sink.db_prefix = sth_
> cygnus-ngsi.sinks.sth-sink.collection_prefix = sth_
> cygnus-ngsi.sinks.sth-sink.resolutions = day,hour,minute
> #cygnus-ngsi.sinks.sth-sink.batch_size = 1
> #cygnus-ngsi.sinks.sth-sink.batch_timeout = 30
> #cygnus-ngsi.sinks.sth-sink.batch_ttl = 10
> #cygnus-ngsi.sinks.sth-sink.data_expiration = 0
> #cygnus-ngsi.sinks.sth-sink.ignore_white_spaces = true
> cygnus-ngsi.channels.sth-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
> cygnus-ngsi.channels.sth-channel.capacity = 1000
> cygnus-ngsi.channels.sth-channel.transactionCapacity = 100
> In the STH side I have the following conf:
> var config = {};
> // STH server configuration
> //--------------------------
> config.server = {
> host: '10.0.2.15',
> port: '8666',
> defaultService: 'testservice',
> defaultServicePath: '/testservicepath',
> filterOutEmpty: 'true',
> aggregationBy: ['day', 'hour', 'minute'],
> temporalDir: 'temp'
> };
> // Database configuration
> //------------------------
> config.database = {
> dataModel: 'collection-per-entity',
> user: '',
> password: '',
> URI: 'localhost:27017',
> replicaSet: '',
> prefix: 'sth_',
> collectionPrefix: 'sth_',
> poolSize: '5',
> shouldStore: 'both',
> truncation: {
> expireAfterSeconds: '0',
> size: '0',
> max: '0'
> },
> ignoreBlankSpaces: 'true',
> nameMapping: {
> enabled: 'false',
> configFile: './name-mapping.json'
> },
> nameEncoding: 'false'
> };
> // Logging configuration
> //------------------------
> config.logging = {
> level: 'info',
> NODE_ENV variable is set to 'development'.
> format: 'pipe',
> proofOfLifeInterval: '60'
> };
> module.exports = config;
--
This message was sent by Atlassian JIRA
(v6.4.1#64016)
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy