Dear Carlos,
Fiware-Service and Fiware-ServicePath headers are important when you want to use Cygnus with a Database. They are the name of table in the database.
Best regards,
Pasquale
-------------------------------------------------------------------------------
Pasquale Vitale created HELC-1025:
-------------------------------------
Summary: [Fiware-finodex-coaching] Problems with COSMOS persistance
Key: HELC-1025
URL: https://jira.fiware.org/browse/HELC-1025
Project: Help-Coaches-Desk
Issue Type: extRequest
Components: FINODEX
Reporter: FW External User
Assignee: Pasquale Vitale
Hi everyone,
I'm Carlos Corrales, part of the development team at SAPIN (FINODEX012). We
have been working hard in the application and now that it's finished and we
are doing tests with final users, we are improving our code. A couple of
months ago we were able to write data to our HDFS in the COSMOS Fiware
machine (cosmos.lab.fi-ware.org) with our user (idelgado). As we are not
using Cosmos for production right now (it will be used in the future), we
left it aside. Now that we are checking that everything works as expected,
we can see that our Cygnus connector is not able to write properly in HDFS.
We haven't changed anything in the configuration, but we receive the
following error when trying to write (flume.log file)
time=2015-09-13T15:11:36.744CEST | lvl=INFO |
trans=1442149853-658-0000000000 | function=persist | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink[367] :
[hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
(idelgado_SAPINnull), Data
({"recvTime":"2015-09-13T15:11:00.413","age":"25",
"age_md":[],"exercisetype":"Light", "exercisetype_md":[],"hr":"36",
"hr_md":[],"id_session":"1234", "id_session_md":[]})
time=2015-09-13T15:11:36.803CEST | lvl=ERROR |
trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] :
Persistence error (The unknownt/idelgado_SAPINnull directory could not be
created in HDFS. HttpFS response: 401 Unauthorized)
time=2015-09-13T15:11:36.804CEST | lvl=INFO |
trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[150] : An event
was put again in the channel (id=1444248959, ttl=0)
time=2015-09-13T15:11:36.804CEST | lvl=INFO |
trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[175] : Finishing
transaction (1442149853-658-0000000000)
As you can see, the error is 401 Unauthorized. However, we can access the
Cosmos machine using SSH, and the user/password in the cygnus.conf file. We
also checked that the IP address of the Cosmos machine hasn't changed.
Could you please help us? We don't know what else to check...
Best regards
--
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching at lists.fiware.org
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
[Created via e-mail received from: Carlos Corrales <carlos at secmotic.com>]
-------------------------------------------------------------------------------
Comments:
Juanjo Hierro - Sunday 4:32 PM
Hi,
I copy Francisco (Paco) Romero who should be able to provide some help on this. I guess he won't be able to work on the matter until tomorrow since support is not granted out of office working hours.
Nevertheless, I believe that it is rather important that you open an issue so that a ticket on our help-desk system is created and then follow-up of support is feasible.
You may create a ticket in our helpdesk simply by sending an email to either fiware-lab-help (if it happens to be an issue which is mostly related to FIWARE Lab or the global COSMOS instance setup on the FIWARE Lab) or fiware-tech-help (if it happens to be an issue mostly related with the technology, i.e., with COSMOS in general and not the particular instance of COSMOS running on the FIWARE Lab).
Best regards,
-- Juanjo
------------------
FW External User - Sunday 4:38 PM
Hi Juanjo,
Thanks for you feedback, we sent it today because we wanted you to answer
it tomorrow morning :).
As you can see, a ticket has been created in Jira (
https://jira.fiware.org/browse/HELC-1025). I used the
fiware-finodex-coaching list, because we are a FINODEX project and they
told us to do so. Should we create another ticket in the fiware-tech-help
list also?
Best regards
--
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching at lists.fiware.org
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
------------------
FW External User - Sunday 4:42 PM
Hi Carlos,
I do not know if this will help. The same thing happened to us.
http://stackoverflow.com/questions/31310111/oauth2-in-cygnus/31312196#31312196
Regards
firma
*SensoWave
Ignacio Gómez Maqueda*
*
*
Director Técnico
móvil: + 34 644350613
__imaqueda_ at sensowave.es <mailto:candres at sensowave.es>_
@queronea
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching at lists.fiware.org
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
------------------
Juanjo Hierro - Yesterday 12:39 AM
I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)
Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.
You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.
Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.
Best regards,
-- Juanjo
------------------
Stefano De Panfilis - Yesterday 12:46 AM
dear carlos,
in fact, as an sme working in finodex you did correctly.
it will be your fiware coach (pasquale vitale in cc) that if not able to
solve himself the issue, and i guess this time he will not, he will turn
the ticket to the ge owner (francisco romero as indicated by juanjo).
ciao,
stefano
--
Stefano De Panfilis
Chief Innovation Officer
Engineering Ingegneria Informatica S.p.A.
via Riccardo Morandi 32
00148 Roma
Italy
tel (direct): +39-06-8759-4253
tel (secr.): +39-068307-4513
fax: +39-068307-4200
cell: +39-335-7542-567
skype: depa01
twitter: @depa01
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching at lists.fiware.org
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
------------------
Pasquale Vitale - Yesterday 2:20 PM
see the helc 1026
------------------
FW External User - Yesterday 6:24 PM
Hi everyone,
Thanks for your responses, we have been testing the whole day and trying to
find out the issue. It is not a problem with neither the password, nor the
oauth2 token. We are able to access using ssh user/password and the curl
command proposed by Pasquale with Oauth2 token. This is what I received
using the Oauth2 token curl proposed by Pasquale
curl -X GET "
http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado"
-H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
{"FileStatuses":{"FileStatus":[{"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0},{"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0},{"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0},{"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0},{"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}]}}
Everything seems to be correct. However, using the same token and updating
ORION, I receive the following error in Cygnus:
time=2015-09-14T18:10:44.419CEST | lvl=ERROR |
trans=1442246992-282-0000000000 | function=process | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] :
Persistence error (The org42/idelgado_tablanull directory could not be
created in HDFS. HttpFS response: 401 Unauthorized)
time=2015-09-14T18:10:44.419CEST | lvl=WARN |
trans=1442246992-282-0000000000 | function=process | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event
TTL has expired, it is no more re-injected in the channel (id=1444248959,
ttl=0)
However, if I try to access the directory shown in the error, I'm able to
do so
ssh idelgado at cosmos.lab.fi-ware.org
hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
Found 1 items
-rw-r----- 3 idelgado idelgado 1416 2015-07-01 13:11
/user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt
My cygnus.conf file content is:
#
============================================
# OrionHDFSSink
configuration
# channel name from where to read notification
events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be
changed
cygnusagent.sinks.hdfs-sink.type =
es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
# Comma-separated list of FQDN/IP address regarding the Cosmos Namenode
endpoints
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000
for httpfs, 50070 for webhdfs and free c\
hoice for
inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
# default username allowed to write in
HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado
# default password for the default
username
#cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i
# HDFS backend type (webhdfs, httpfs or
infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
# how the attributes are stored, either per row either per column (row,
column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column
# prefix for the database and table names, empty if no prefix is
desired
cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla
# Hive FQDN/IP address of the Hive
server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
# Hive port for Hive external table
provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
Could you please help us?
Thanks in advance!
2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <
--
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching at lists.fiware.org
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
------------------
FW External User - Yesterday 6:31 PM
One more thing, Francisco, how can I find the Cygnus version?
Best regards
--
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching at lists.fiware.org
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
------------------
FW External User - Yesterday 6:59 PM
Hi everyone again,
Could it be related to the Fiware-Service and Fiware-ServicePath headers?
We didn't use them at the beginning, then we tried to use them, and now, we
notice that if we didn't use them the directory used by Cygnus to write in
Cosmos is /org42/idelgado_tablanull, while if we use for instance
--header 'Fiware-Service: unknownT' --header 'Fiware-ServicePath: /'
the directory written is /unknownt/idelgado_tablanull.
Any clues?
Best regards
--
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching at lists.fiware.org
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
------------------
--
This email was generated by the JIRA Email This Issue plugin (https://marketplace.atlassian.com/plugins/com.metainf.jira.plugin.emailissue) from www.meta-inf.hu (http://www.meta-inf.hu).
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy