Be aware there exist this issue in JIRA for this exchange: https://jira.fiware.org/browse/HELC-1025
From: FRANCISCO ROMERO BUENO
Sent: martes, 15 de septiembre de 2015 9:12
To: Carlos Corrales; MANUEL ESCRICHE VICENTE
Cc: JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching at lists.fi-ware.org; jose at secmotic.com
Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
Hi Carlos,
I will aggregate through this email all the other emails you sent.
Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing:
$ rpm -qa | grep cygnus
Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean:
* Current version is 0.8.2
* The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink" part is previous to 0.8.0. From 0.8.0 it must be "com.telefonica.iot.cygnus.sinks.OrionCKANSink"
* The OAuth2 token was introduced in 0.8.2
My recommendation is you install the latest version from RPM (https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized):
https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md
https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md<https://github.com/telefonicaid/fiware-cygnus/tree/release/0.8.2>
Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus' HTTPSource are used. Again, it seems you are not using/configuring these values.
Regards,
Francisco
De: Carlos Corrales <carlos at secmotic.com<mailto:carlos at secmotic.com>>
Fecha: lunes, 14 de septiembre de 2015, 18:22
Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente at telefonica.com<mailto:manuel.escrichevicente at telefonica.com>>
CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro at telefonica.com<mailto:juanjose.hierro at telefonica.com>>, "fiware-finodex-coaching at lists.fi-ware.org<mailto:fiware-finodex-coaching at lists.fi-ware.org>" <fiware-finodex-coaching at lists.fi-ware.org<mailto:fiware-finodex-coaching at lists.fi-ware.org>>, "jose at secmotic.com<mailto:jose at secmotic.com>" <jose at secmotic.com<mailto:jose at secmotic.com>>, Francisco Romero Bueno <francisco.romerobueno at telefonica.com<mailto:francisco.romerobueno at telefonica.com>>
Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
Hi everyone,
Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale
curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
{"FileStatuses":{"FileStatus":[{"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0},{"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0},{"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0},{"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0},{"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}]}}
Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus:
time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized)
time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0)
However, if I try to access the directory shown in the error, I'm able to do so
ssh idelgado at cosmos.lab.fi-ware.org<mailto:idelgado at cosmos.lab.fi-ware.org>
hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
Found 1 items
-rw-r----- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt
My cygnus.conf file content is:
# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
# Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\
hoice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
# default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado
# default password for the default username
#cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i
# HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column
# prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla
# Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
# Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
Could you please help us?
Thanks in advance!
2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente at telefonica.com<mailto:manuel.escrichevicente at telefonica.com>>:
In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request.
In the normal flow GE owners enter into action upon request from Coaches.
Kind regards,
Manuel
From: JUAN JOSE HIERRO SUREDA
Sent: lunes, 14 de septiembre de 2015 0:37
To: Carlos Corrales
Cc: fiware-finodex-coaching at lists.fi-ware.org<mailto:fiware-finodex-coaching at lists.fi-ware.org>; jose at secmotic.com<mailto:jose at secmotic.com>; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)
Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.
You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.
Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.
Best regards,
-- Juanjo
______________________________________________________
Coordinator and Chief Architect, FIWARE platform
CTO Industrial IoT, Telefónica
email: juanjose.hierro at telefonica.com<mailto:juanjose.hierro at telefonica.com>
twitter: @JuanjoHierro
You can follow FIWARE at:
website: http://www.fiware.org
twitter: @FIWARE
facebook: http://www.facebook.com/pages/FI-WARE/251366491587242
linkedIn: http://www.linkedin.com/groups/FIWARE-4239932
On 13/09/15 16:36, Carlos Corrales wrote:
Hi Juanjo,
Thanks for you feedback, we sent it today because we wanted you to answer it tomorrow morning :).
As you can see, a ticket has been created in Jira (https://jira.fiware.org/browse/HELC-1025). I used the fiware-finodex-coaching list, because we are a FINODEX project and they told us to do so. Should we create another ticket in the fiware-tech-help list also?
Best regards
2015-09-13 16:30 GMT+02:00 Juanjo Hierro <juanjose.hierro at telefonica.com<mailto:juanjose.hierro at telefonica.com>>:
Hi,
I copy Francisco (Paco) Romero who should be able to provide some help on this. I guess he won't be able to work on the matter until tomorrow since support is not granted out of office working hours.
Nevertheless, I believe that it is rather important that you open an issue so that a ticket on our help-desk system is created and then follow-up of support is feasible.
You may create a ticket in our helpdesk simply by sending an email to either fiware-lab-help (if it happens to be an issue which is mostly related to FIWARE Lab or the global COSMOS instance setup on the FIWARE Lab) or fiware-tech-help (if it happens to be an issue mostly related with the technology, i.e., with COSMOS in general and not the particular instance of COSMOS running on the FIWARE Lab).
Best regards,
-- Juanjo
______________________________________________________
Coordinator and Chief Architect, FIWARE platform
CTO Industrial IoT, Telefónica
email: juanjose.hierro at telefonica.com<mailto:juanjose.hierro at telefonica.com>
twitter: @JuanjoHierro
You can follow FIWARE at:
website: http://www.fiware.org
twitter: @FIWARE
facebook: http://www.facebook.com/pages/FI-WARE/251366491587242
linkedIn: http://www.linkedin.com/groups/FIWARE-4239932
On 13/09/15 15:20, Carlos Corrales wrote:
Hi everyone,
I'm Carlos Corrales, part of the development team at SAPIN (FINODEX012). We have been working hard in the application and now that it's finished and we are doing tests with final users, we are improving our code. A couple of months ago we were able to write data to our HDFS in the COSMOS Fiware machine (cosmos.lab.fi-ware.org<http://cosmos.lab.fi-ware.org>) with our user (idelgado). As we are not using Cosmos for production right now (it will be used in the future), we left it aside. Now that we are checking that everything works as expected, we can see that our Cygnus connector is not able to write properly in HDFS. We haven't changed anything in the configuration, but we receive the following error when trying to write (flume.log file)
time=2015-09-13T15:11:36.744CEST | lvl=INFO | trans=1442149853-658-0000000000 | function=persist | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink[367] : [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (idelgado_SAPINnull), Data ({"recvTime":"2015-09-13T15:11:00.413","age":"25", "age_md":[],"exercisetype":"Light", "exercisetype_md":[],"hr":"36", "hr_md":[],"id_session":"1234", "id_session_md":[]})
time=2015-09-13T15:11:36.803CEST | lvl=ERROR | trans=1442149853-658-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] : Persistence error (The unknownt/idelgado_SAPINnull directory could not be created in HDFS. HttpFS response: 401 Unauthorized)
time=2015-09-13T15:11:36.804CEST | lvl=INFO | trans=1442149853-658-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[150] : An event was put again in the channel (id=1444248959, ttl=0)
time=2015-09-13T15:11:36.804CEST | lvl=INFO | trans=1442149853-658-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[175] : Finishing transaction (1442149853-658-0000000000)
As you can see, the error is 401 Unauthorized. However, we can access the Cosmos machine using SSH, and the user/password in the cygnus.conf file. We also checked that the IP address of the Cosmos machine hasn't changed.
Could you please help us? We don't know what else to check...
Best regards
--
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
[https://docs.google.com/uc?export=download&id=0B02xdGS72R1tVWItcDVEUDlvWmM&revid=0B02xdGS72R1tdWYzTzcrMXZ3U3ArSndLUEl0eVB6ZUFtOTJrPQ]
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching at lists.fiware.org<mailto:Fiware-finodex-coaching at lists.fiware.org>
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
________________________________
Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
--
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
[https://docs.google.com/uc?export=download&id=0B02xdGS72R1tVWItcDVEUDlvWmM&revid=0B02xdGS72R1tdWYzTzcrMXZ3U3ArSndLUEl0eVB6ZUFtOTJrPQ]
________________________________
Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
--
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
[https://docs.google.com/uc?export=download&id=0B02xdGS72R1tVWItcDVEUDlvWmM&revid=0B02xdGS72R1tdWYzTzcrMXZ3U3ArSndLUEl0eVB6ZUFtOTJrPQ]
________________________________
Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-finodex-coaching/attachments/20150915/17f2ba71/attachment.html>
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy