[Backlogmanager] [FIWARE-JIRA] (HELP-15545) [fiware-stackoverflow] FIWARE Cygnus: File exists in HDFS, but could not write the data

[Fernando Lopez (JIRA)]

     [ https://jira.fiware.org/browse/HELP-15545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fernando Lopez updated HELP-15545:
----------------------------------
    Description: 
Created question in FIWARE Q/A platform on 07-03-2019 at 09:03
{color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/55039350/fiware-cygnus-file-exists-in-hdfs-but-could-not-write-the-data


+Question:+
FIWARE Cygnus: File exists in HDFS, but could not write the data

+Description:+
I am using NGSIHDFSSink to persist on an HDFS through Cygnus. When the first notification from Orion is received, it creates the directory and the correct file with the data:

/user/hdfs/openiot/Lamp_001_Lamp/Lamp_001_Lamp.txt


However, for the next notifications received it shows the following error:


  There was some problem with the current endpoint, trying other one.
  Details: CygnusPersistenceError.
  /user/hdfs/openioLamp/Lamp_001_Lamp.txt file exists in HDFS, but could
  not write the data. Server response: 403 Forbidden.


This is my Sink configuration:

cygnus-ngsi.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.NGSIHDFSSink
cygnus-ngsi.sinks.hdfs-sink.channel = hdfs-channel
cygnus-ngsi.sinks.hdfs-sink.enable_encoding = false
cygnus-ngsi.sinks.hdfs-sink.enable_grouping = false
cygnus-ngsi.sinks.hdfs-sink.enable_lowercase = false
cygnus-ngsi.sinks.hdfs-sink.enable_name_mappings = false
cygnus-ngsi.sinks.hdfs-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.hdfs-sink.file_format = json-column
cygnus-ngsi.sinks.hdfs-sink.backend.impl = rest
cygnus-ngsi.sinks.hdfs-sink.backend.max_conns = 500
cygnus-ngsi.sinks.hdfs-sink.backend.max_conns_per_route = 100
cygnus-ngsi.sinks.hdfs-sink.hdfs_host = my_hdfs_ip
cygnus-ngsi.sinks.hdfs-sink.hdfs_port = 50070
cygnus-ngsi.sinks.hdfs-sink.hdfs_username = hdfs
cygnus-ngsi.sinks.hdfs-sink.hdfs_password =
cygnus-ngsi.sinks.hdfs-sink.oauth2_token =
cygnus-ngsi.sinks.hdfs-sink.service_as_namespace = false
cygnus-ngsi.sinks.hdfs-sink.batch_size = 100
cygnus-ngsi.sinks.hdfs-sink.batch_timeout = 30
cygnus-ngsi.sinks.hdfs-sink.batch_ttl = 10
cygnus-ngsi.sinks.hdfs-sink.batch_retry_intervals = 5000
cygnus-ngsi.sinks.hdfs-sink.hive = false
cygnus-ngsi.sinks.hdfs-sink.krb5_auth = false


Any ideas?

Thank you so much


  was:

Created question in FIWARE Q/A platform on 07-03-2019 at 09:03
{color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/55039350/fiware-cygnus-file-exists-in-hdfs-but-could-not-write-the-data


+Question:+
FIWARE Cygnus: File exists in HDFS, but could not write the data

+Description:+
I am using NGSIHDFSSink to persist on an HDFS through Cygnus. When the first notification from Orion is received, it creates the directory and the correct file with the data:

/user/hdfs/openiot/Lamp_001_Lamp/Lamp_001_Lamp.txt


However, for the next notifications received it shows the following error:


  There was some problem with the current endpoint, trying other one.
  Details: CygnusPersistenceError.
  /user/hdfs/openioLamp/Lamp_001_Lamp.txt file exists in HDFS, but could
  not write the data. Server response: 403 Forbidden.


This is my Sink configuration:

cygnus-ngsi.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.NGSIHDFSSink
cygnus-ngsi.sinks.hdfs-sink.channel = hdfs-channel
cygnus-ngsi.sinks.hdfs-sink.enable_encoding = false
cygnus-ngsi.sinks.hdfs-sink.enable_grouping = false
cygnus-ngsi.sinks.hdfs-sink.enable_lowercase = false
cygnus-ngsi.sinks.hdfs-sink.enable_name_mappings = false
cygnus-ngsi.sinks.hdfs-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.hdfs-sink.file_format = json-column
cygnus-ngsi.sinks.hdfs-sink.backend.impl = rest
cygnus-ngsi.sinks.hdfs-sink.backend.max_conns = 500
cygnus-ngsi.sinks.hdfs-sink.backend.max_conns_per_route = 100
cygnus-ngsi.sinks.hdfs-sink.hdfs_host = my_hdfs_ip
cygnus-ngsi.sinks.hdfs-sink.hdfs_port = 50070
cygnus-ngsi.sinks.hdfs-sink.hdfs_username = hdfs
cygnus-ngsi.sinks.hdfs-sink.hdfs_password =
cygnus-ngsi.sinks.hdfs-sink.oauth2_token =
cygnus-ngsi.sinks.hdfs-sink.service_as_namespace = false
cygnus-ngsi.sinks.hdfs-sink.batch_size = 100
cygnus-ngsi.sinks.hdfs-sink.batch_timeout = 30
cygnus-ngsi.sinks.hdfs-sink.batch_ttl = 10
cygnus-ngsi.sinks.hdfs-sink.batch_retry_intervals = 5000
cygnus-ngsi.sinks.hdfs-sink.hive = false
cygnus-ngsi.sinks.hdfs-sink.krb5_auth = false


Any ideas?

Thank you so much


     HD-Enabler: Cygnus

> [fiware-stackoverflow] FIWARE Cygnus: File exists in HDFS, but could not write the data
> ---------------------------------------------------------------------------------------
>
>                 Key: HELP-15545
>                 URL: https://jira.fiware.org/browse/HELP-15545
>             Project: Help-Desk
>          Issue Type: Monitor
>          Components: FIWARE-TECH-HELP
>            Reporter: Backlog Manager
>              Labels: fiware, fiware-cygnus, hadoop, hdfs
>
> Created question in FIWARE Q/A platform on 07-03-2019 at 09:03
> {color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/55039350/fiware-cygnus-file-exists-in-hdfs-but-could-not-write-the-data
> +Question:+
> FIWARE Cygnus: File exists in HDFS, but could not write the data
> +Description:+
> I am using NGSIHDFSSink to persist on an HDFS through Cygnus. When the first notification from Orion is received, it creates the directory and the correct file with the data:
> /user/hdfs/openiot/Lamp_001_Lamp/Lamp_001_Lamp.txt
> However, for the next notifications received it shows the following error:
>   There was some problem with the current endpoint, trying other one.
>   Details: CygnusPersistenceError.
>   /user/hdfs/openioLamp/Lamp_001_Lamp.txt file exists in HDFS, but could
>   not write the data. Server response: 403 Forbidden.
> This is my Sink configuration:
> cygnus-ngsi.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.NGSIHDFSSink
> cygnus-ngsi.sinks.hdfs-sink.channel = hdfs-channel
> cygnus-ngsi.sinks.hdfs-sink.enable_encoding = false
> cygnus-ngsi.sinks.hdfs-sink.enable_grouping = false
> cygnus-ngsi.sinks.hdfs-sink.enable_lowercase = false
> cygnus-ngsi.sinks.hdfs-sink.enable_name_mappings = false
> cygnus-ngsi.sinks.hdfs-sink.data_model = dm-by-entity
> cygnus-ngsi.sinks.hdfs-sink.file_format = json-column
> cygnus-ngsi.sinks.hdfs-sink.backend.impl = rest
> cygnus-ngsi.sinks.hdfs-sink.backend.max_conns = 500
> cygnus-ngsi.sinks.hdfs-sink.backend.max_conns_per_route = 100
> cygnus-ngsi.sinks.hdfs-sink.hdfs_host = my_hdfs_ip
> cygnus-ngsi.sinks.hdfs-sink.hdfs_port = 50070
> cygnus-ngsi.sinks.hdfs-sink.hdfs_username = hdfs
> cygnus-ngsi.sinks.hdfs-sink.hdfs_password =
> cygnus-ngsi.sinks.hdfs-sink.oauth2_token =
> cygnus-ngsi.sinks.hdfs-sink.service_as_namespace = false
> cygnus-ngsi.sinks.hdfs-sink.batch_size = 100
> cygnus-ngsi.sinks.hdfs-sink.batch_timeout = 30
> cygnus-ngsi.sinks.hdfs-sink.batch_ttl = 10
> cygnus-ngsi.sinks.hdfs-sink.batch_retry_intervals = 5000
> cygnus-ngsi.sinks.hdfs-sink.hive = false
> cygnus-ngsi.sinks.hdfs-sink.krb5_auth = false
> Any ideas?
> Thank you so much



--
This message was sent by Atlassian JIRA
(v6.4.1#64016)