[ https://jira.fiware.org/browse/HELP-9159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fernando Lopez updated HELP-9159:
---------------------------------
HD-Chapter: Data
Description:
Created question in FIWARE Q/A platform on 08-07-2015 at 13:07
{color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/31291365/null-name-is-given-to-both-folder-and-file-when-sinking-data-from-orion-to-cos
+Question:+
"null" name is given to both folder and file when sinking data from Orion to Cosmos using Cygnus
+Description:+
I have an issue related to ngsi2cosmos data flow. Everything works fine when persisting the information received in Orion into the public instance of Cosmos, but the destination folder and file name are both "null".
Simple test as follows:
I create of a brand new NGSIEntity with these headers added: Fiware-Service: myservice & Fiware-ServicePath: /my
I add a new subscription with Cygnus as the reference endpoint.
I send an update to previously created NGSIEntity
When I check my user space in Cosmos I check that the following route has been created: /user/myuser/myservice/null/null.txt
File content is OK, every updated info in Orion has been correctly sinked into it. The problem is with folder and file names.
I can't make it work properly. Isn't it supposed to get entityId and entityType for folder and file naming?
Component versions:
Orion version: contextBroker-0.19.0-1.x86_64
Cygnus version: cygnus-0.5-91.g3eb100e.x86_64
Cosmos: global instance
Cygnus conf file:
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default organization (organization semantic depend on the persistence sink)
cygnusagent.sources.http-source.handler.default_organization = org42
# Number of channel re-injection retries before a Flume event is definitely discarded
cygnusagent.sources.http-source.handler.events_ttl = 10
# Management interface port (FIXME: temporal location for this parameter)
cygnusagent.sources.http-source.handler.management_port = 8081
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts
# Timestamp interceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# Destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwreconnectors.cygnus.interceptors.DestinationExtractor$Builder
# Matching table for the destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.matching_table = matching_table.conf
# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
# Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
# default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = myuser
# default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = mypassword
# HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column
# prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix =
# Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
# Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
#=============================================
# hdfs-channel configuration
# channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory
# capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
was:
Created question in FIWARE Q/A platform on 08-07-2015 at 13:07
{color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/31291365/null-name-is-given-to-both-folder-and-file-when-sinking-data-from-orion-to-cos
+Question:+
"null" name is given to both folder and file when sinking data from Orion to Cosmos using Cygnus
+Description:+
I have an issue related to ngsi2cosmos data flow. Everything works fine when persisting the information received in Orion into the public instance of Cosmos, but the destination folder and file name are both "null".
Simple test as follows:
I create of a brand new NGSIEntity with these headers added: Fiware-Service: myservice & Fiware-ServicePath: /my
I add a new subscription with Cygnus as the reference endpoint.
I send an update to previously created NGSIEntity
When I check my user space in Cosmos I check that the following route has been created: /user/myuser/myservice/null/null.txt
File content is OK, every updated info in Orion has been correctly sinked into it. The problem is with folder and file names.
I can't make it work properly. Isn't it supposed to get entityId and entityType for folder and file naming?
Component versions:
Orion version: contextBroker-0.19.0-1.x86_64
Cygnus version: cygnus-0.5-91.g3eb100e.x86_64
Cosmos: global instance
Cygnus conf file:
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default organization (organization semantic depend on the persistence sink)
cygnusagent.sources.http-source.handler.default_organization = org42
# Number of channel re-injection retries before a Flume event is definitely discarded
cygnusagent.sources.http-source.handler.events_ttl = 10
# Management interface port (FIXME: temporal location for this parameter)
cygnusagent.sources.http-source.handler.management_port = 8081
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts
# Timestamp interceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# Destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwreconnectors.cygnus.interceptors.DestinationExtractor$Builder
# Matching table for the destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.matching_table = matching_table.conf
# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
# Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
# default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = myuser
# default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = mypassword
# HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column
# prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix =
# Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
# Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
#=============================================
# hdfs-channel configuration
# channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory
# capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
HD-Enabler: Cosmos
> [fiware-stackoverflow] "null" name is given to both folder and file when sinking data from Orion to Cosmos using Cygnus
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: HELP-9159
> URL: https://jira.fiware.org/browse/HELP-9159
> Project: Help-Desk
> Issue Type: Monitor
> Components: FIWARE-TECH-HELP
> Reporter: Backlog Manager
> Labels: cosmos, fiware, fiware-cosmos, fiware-cygnus
>
> Created question in FIWARE Q/A platform on 08-07-2015 at 13:07
> {color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/31291365/null-name-is-given-to-both-folder-and-file-when-sinking-data-from-orion-to-cos
> +Question:+
> "null" name is given to both folder and file when sinking data from Orion to Cosmos using Cygnus
> +Description:+
> I have an issue related to ngsi2cosmos data flow. Everything works fine when persisting the information received in Orion into the public instance of Cosmos, but the destination folder and file name are both "null".
> Simple test as follows:
> I create of a brand new NGSIEntity with these headers added: Fiware-Service: myservice & Fiware-ServicePath: /my
> I add a new subscription with Cygnus as the reference endpoint.
> I send an update to previously created NGSIEntity
> When I check my user space in Cosmos I check that the following route has been created: /user/myuser/myservice/null/null.txt
> File content is OK, every updated info in Orion has been correctly sinked into it. The problem is with folder and file names.
> I can't make it work properly. Isn't it supposed to get entityId and entityType for folder and file naming?
> Component versions:
> Orion version: contextBroker-0.19.0-1.x86_64
> Cygnus version: cygnus-0.5-91.g3eb100e.x86_64
> Cosmos: global instance
> Cygnus conf file:
> cygnusagent.sources = http-source
> cygnusagent.sinks = hdfs-sink
> cygnusagent.channels = hdfs-channel
> #=============================================
> # source configuration
> # channel name where to write the notification events
> cygnusagent.sources.http-source.channels = hdfs-channel
> # source class, must not be changed
> cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
> # listening port the Flume source will use for receiving incoming notifications
> cygnusagent.sources.http-source.port = 5050
> # Flume handler that will parse the notifications, must not be changed
> cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
> # URL target
> cygnusagent.sources.http-source.handler.notification_target = /notify
> # Default organization (organization semantic depend on the persistence sink)
> cygnusagent.sources.http-source.handler.default_organization = org42
> # Number of channel re-injection retries before a Flume event is definitely discarded
> cygnusagent.sources.http-source.handler.events_ttl = 10
> # Management interface port (FIXME: temporal location for this parameter)
> cygnusagent.sources.http-source.handler.management_port = 8081
> # Source interceptors, do not change
> cygnusagent.sources.http-source.interceptors = ts
> # Timestamp interceptor, do not change
> cygnusagent.sources.http-source.interceptors.ts.type = timestamp
> # Destination extractor interceptor, do not change
> cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwreconnectors.cygnus.interceptors.DestinationExtractor$Builder
> # Matching table for the destination extractor interceptor, do not change
> cygnusagent.sources.http-source.interceptors.de.matching_table = matching_table.conf
> # ============================================
> # OrionHDFSSink configuration
> # channel name from where to read notification events
> cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
> # sink class, must not be changed
> cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
> # Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
> cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
> # port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
> cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
> # default username allowed to write in HDFS
> cygnusagent.sinks.hdfs-sink.cosmos_default_username = myuser
> # default password for the default username
> cygnusagent.sinks.hdfs-sink.cosmos_default_password = mypassword
> # HDFS backend type (webhdfs, httpfs or infinity)
> cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
> # how the attributes are stored, either per row either per column (row, column)
> cygnusagent.sinks.hdfs-sink.attr_persistence = column
> # prefix for the database and table names, empty if no prefix is desired
> cygnusagent.sinks.hdfs-sink.naming_prefix =
> # Hive FQDN/IP address of the Hive server
> cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
> # Hive port for Hive external table provisioning
> cygnusagent.sinks.hdfs-sink.hive_port = 10000
> #=============================================
> # hdfs-channel configuration
> # channel type (must not be changed)
> cygnusagent.channels.hdfs-channel.type = memory
> # capacity of the channel
> cygnusagent.channels.hdfs-channel.capacity = 1000
> # amount of bytes that can be sent per transaction
> cygnusagent.channels.hdfs-channel.transactionCapacity = 100
--
This message was sent by Atlassian JIRA
(v6.4.1#64016)
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy