[Fiware-finodex-coaching] Problems with COSMOS persistance

Carlos Corrales carlos at secmotic.com
Mon Sep 14 18:58:00 CEST 2015


Hi everyone again,

Could it be related to the Fiware-Service and Fiware-ServicePath headers?
We didn't use them at the beginning, then we tried to use them, and now, we
notice that if we didn't use them the directory used by Cygnus to write in
Cosmos is /org42/idelgado_tablanull, while if we use for instance

--header 'Fiware-Service: unknownT' --header 'Fiware-ServicePath: /'

the directory written is /unknownt/idelgado_tablanull.

Any clues?

Best regards


2015-09-14 18:29 GMT+02:00 Carlos Corrales <carlos at secmotic.com>:

> One more thing, Francisco, how can I find the Cygnus version?
>
>
> Best regards
>
> 2015-09-14 18:22 GMT+02:00 Carlos Corrales <carlos at secmotic.com>:
>
>> Hi everyone,
>>
>> Thanks for your responses, we have been testing the whole day and trying
>> to find out the issue. It is not a problem with neither the password, nor
>> the oauth2 token. We are able to access using ssh user/password and the
>> curl command proposed by Pasquale with Oauth2 token. This is what I
>> received using the Oauth2 token curl proposed by Pasquale
>>
>> curl -X GET  "
>> http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado"
>> -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
>>
>> {"FileStatuses":{"FileStatus":[{"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0},{"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0},{"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0},{"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0},{"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}]}}
>>
>>
>> Everything seems to be correct. However, using the same token and
>> updating ORION, I receive the following error in Cygnus:
>>
>> time=2015-09-14T18:10:44.419CEST | lvl=ERROR |
>> trans=1442246992-282-0000000000 | function=process | comp=Cygnus |
>> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] :
>> Persistence error (The org42/idelgado_tablanull directory could not be
>> created in HDFS. HttpFS response: 401 Unauthorized)
>> time=2015-09-14T18:10:44.419CEST | lvl=WARN |
>> trans=1442246992-282-0000000000 | function=process | comp=Cygnus |
>> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event
>> TTL has expired, it is no more re-injected in the channel (id=1444248959,
>> ttl=0)
>>
>> However, if I try to access the directory shown in the error, I'm able to
>> do so
>>
>> ssh idelgado at cosmos.lab.fi-ware.org
>> hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
>> Found 1 items
>> -rw-r-----   3 idelgado idelgado       1416 2015-07-01 13:11
>> /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt
>>
>> My cygnus.conf file content is:
>>
>> #
>> ============================================
>>
>> # OrionHDFSSink
>> configuration
>>
>> # channel name from where to read notification
>> events
>> cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
>> # sink class, must not be
>> changed
>>
>> cygnusagent.sinks.hdfs-sink.type =
>> es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
>> # Comma-separated list of FQDN/IP address regarding the Cosmos Namenode
>> endpoints
>> cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
>> # port of the Cosmos service listening for persistence operations; 14000
>> for httpfs, 50070 for webhdfs and free c\
>> hoice for
>> inifinty
>>
>> cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
>> # default username allowed to write in
>> HDFS
>> cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado
>> # default password for the default
>> username
>>
>> #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
>>
>> cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i
>> # HDFS backend type (webhdfs, httpfs or
>> infinity)
>> cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
>> # how the attributes are stored, either per row either per column (row,
>> column)
>> cygnusagent.sinks.hdfs-sink.attr_persistence = column
>> # prefix for the database and table names, empty if no prefix is
>> desired
>> cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla
>> # Hive FQDN/IP address of the Hive
>> server
>>
>> cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
>> # Hive port for Hive external table
>> provisioning
>>
>> cygnusagent.sinks.hdfs-sink.hive_port = 10000
>>
>> Could you please help us?
>>
>> Thanks in advance!
>>
>>
>>
>>
>>
>> 2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <
>> manuel.escrichevicente at telefonica.com>:
>>
>>> In this case I see both Pasquale (Coach) and Francisco (GE Owner) have
>>> reacted to the request.
>>>
>>> In the normal flow GE owners enter into action upon request from
>>> Coaches.
>>>
>>> Kind regards,
>>>
>>> Manuel
>>>
>>>
>>>
>>> *From:* JUAN JOSE HIERRO SUREDA
>>> *Sent:* lunes, 14 de septiembre de 2015 0:37
>>> *To:* Carlos Corrales
>>> *Cc:* fiware-finodex-coaching at lists.fi-ware.org; jose at secmotic.com;
>>> FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
>>> *Subject:* Re: [Fiware-finodex-coaching] Problems with COSMOS
>>> persistance
>>>
>>>
>>>
>>>
>>>   I guess that the FINODEX coach will forward the ticket to the FIWARE
>>> GE owner (or eventually, if he hadn't know where to redirect it, he would
>>> redirect the original question to the fiware-tech-help mailing list)
>>>
>>>   Point is that one way or the other, it would arrive at the inbox of
>>> the FIWARE GE owner who has to take care of the ticket.
>>>
>>>   You may have sent the ticket directly to the fiware-tech-help mailing
>>> list ... however, I guess that the FINODEX coach team wanted to be informed
>>> to keep track of the evolution.
>>>
>>>   Manuel, in copy, is the master of all the helpdesk processes and he
>>> can clarify what the actual workflow is.
>>>
>>>   Best regards,
>>>
>>> -- Juanjo
>>>
>>> ______________________________________________________
>>>
>>>
>>>
>>> Coordinator and Chief Architect, FIWARE platform
>>>
>>> CTO Industrial IoT, Telefónica
>>>
>>>
>>>
>>> email: juanjose.hierro at telefonica.com
>>>
>>> twitter: @JuanjoHierro
>>>
>>>
>>>
>>> You can follow FIWARE at:
>>>
>>>   website:  http://www.fiware.org
>>>
>>>   twitter:  @FIWARE
>>>
>>>   facebook: http://www.facebook.com/pages/FI-WARE/251366491587242
>>>
>>>   linkedIn: http://www.linkedin.com/groups/FIWARE-4239932
>>>
>>> On 13/09/15 16:36, Carlos Corrales wrote:
>>>
>>> Hi Juanjo,
>>>
>>> Thanks for you feedback, we sent it today because we wanted you to
>>> answer it tomorrow morning :).
>>>
>>>
>>>
>>> As you can see, a ticket has been created in Jira (
>>> https://jira.fiware.org/browse/HELC-1025). I used the
>>> fiware-finodex-coaching list, because we are a FINODEX project and they
>>> told us to do so. Should we create another ticket in the fiware-tech-help
>>> list also?
>>>
>>> Best regards
>>>
>>>
>>>
>>> 2015-09-13 16:30 GMT+02:00 Juanjo Hierro <juanjose.hierro at telefonica.com
>>> >:
>>>
>>> Hi,
>>>
>>>   I copy Francisco (Paco) Romero who should be able to provide some help
>>> on this.   I guess he won't be able to work on the matter until tomorrow
>>> since support is not granted out of office working hours.
>>>
>>>   Nevertheless, I believe that it is rather important that you open an
>>> issue so that a ticket on our help-desk system is created and then
>>> follow-up of support is feasible.
>>>
>>>   You may create a ticket in our helpdesk simply by sending an email to
>>> either fiware-lab-help (if it happens to be an issue which is mostly
>>> related to FIWARE Lab or the global COSMOS instance setup on the FIWARE
>>> Lab) or fiware-tech-help (if it happens to be an issue mostly related with
>>> the technology, i.e., with COSMOS in general and not the particular
>>> instance of COSMOS running on the FIWARE Lab).
>>>
>>>   Best regards,
>>>
>>> -- Juanjo
>>>
>>> ______________________________________________________
>>>
>>>
>>>
>>> Coordinator and Chief Architect, FIWARE platform
>>>
>>> CTO Industrial IoT, Telefónica
>>>
>>>
>>>
>>> email: juanjose.hierro at telefonica.com
>>>
>>> twitter: @JuanjoHierro
>>>
>>>
>>>
>>> You can follow FIWARE at:
>>>
>>>   website:  http://www.fiware.org
>>>
>>>   twitter:  @FIWARE
>>>
>>>   facebook: http://www.facebook.com/pages/FI-WARE/251366491587242
>>>
>>>   linkedIn: http://www.linkedin.com/groups/FIWARE-4239932
>>>
>>> On 13/09/15 15:20, Carlos Corrales wrote:
>>>
>>> Hi everyone,
>>>
>>> I'm Carlos Corrales, part of the development team at SAPIN (FINODEX012).
>>> We have been working hard in the application and now that it's finished and
>>> we are doing tests with final users, we are improving our code. A couple of
>>> months ago we were able to write data to our HDFS in the COSMOS Fiware
>>> machine (cosmos.lab.fi-ware.org) with our user (idelgado). As we are
>>> not using Cosmos for production right now (it will be used in the future),
>>> we left it aside. Now that we are checking that everything works as
>>> expected, we can see that our Cygnus connector is not able to write
>>> properly in HDFS. We haven't changed anything in the configuration, but we
>>> receive the following error when trying to write (flume.log file)
>>>
>>> time=2015-09-13T15:11:36.744CEST | lvl=INFO |
>>> trans=1442149853-658-0000000000 | function=persist | comp=Cygnus |
>>> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink[367] :
>>> [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
>>> (idelgado_SAPINnull), Data
>>> ({"recvTime":"2015-09-13T15:11:00.413","age":"25",
>>> "age_md":[],"exercisetype":"Light", "exercisetype_md":[],"hr":"36",
>>> "hr_md":[],"id_session":"1234", "id_session_md":[]})
>>> time=2015-09-13T15:11:36.803CEST | lvl=ERROR |
>>> trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
>>> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] :
>>> Persistence error (The unknownt/idelgado_SAPINnull directory could not be
>>> created in HDFS. HttpFS response: 401 Unauthorized)
>>> time=2015-09-13T15:11:36.804CEST | lvl=INFO |
>>> trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
>>> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[150] : An event
>>> was put again in the channel (id=1444248959, ttl=0)
>>> time=2015-09-13T15:11:36.804CEST | lvl=INFO |
>>> trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
>>> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[175] : Finishing
>>> transaction (1442149853-658-0000000000)
>>>
>>> As you can see, the error is 401 Unauthorized. However, we can access
>>> the Cosmos machine using SSH, and the user/password in the cygnus.conf
>>> file. We also checked that the IP address of the Cosmos machine hasn't
>>> changed.
>>>
>>> Could you please help us? We don't know what else to check...
>>>
>>> Best regards
>>>
>>>
>>> --
>>>
>>> Carlos Corrales
>>>
>>> Cofounder Secmotic Innovation S.L.
>>>
>>> http://www.secmotic.com
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> Fiware-finodex-coaching mailing list
>>>
>>> Fiware-finodex-coaching at lists.fiware.org
>>>
>>> https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
>>>
>>>
>>> ------------------------------
>>>
>>>
>>> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
>>> puede contener información privilegiada o confidencial y es para uso
>>> exclusivo de la persona o entidad de destino. Si no es usted. el
>>> destinatario indicado, queda notificado de que la lectura, utilización,
>>> divulgación y/o copia sin autorización puede estar prohibida en virtud de
>>> la legislación vigente. Si ha recibido este mensaje por error, le rogamos
>>> que nos lo comunique inmediatamente por esta misma vía y proceda a su
>>> destrucción.
>>>
>>> The information contained in this transmission is privileged and
>>> confidential information intended only for the use of the individual or
>>> entity named above. If the reader of this message is not the intended
>>> recipient, you are hereby notified that any dissemination, distribution or
>>> copying of this communication is strictly prohibited. If you have received
>>> this transmission in error, do not read it. Please immediately reply to the
>>> sender that you have received this communication in error and then delete
>>> it.
>>>
>>> Esta mensagem e seus anexos se dirigem exclusivamente ao seu
>>> destinatário, pode conter informação privilegiada ou confidencial e é para
>>> uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o
>>> destinatário indicado, fica notificado de que a leitura, utilização,
>>> divulgação e/ou cópia sem autorização pode estar proibida em virtude da
>>> legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos
>>> o comunique imediatamente por esta mesma via e proceda a sua destruição
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Carlos Corrales
>>>
>>> Cofounder Secmotic Innovation S.L.
>>>
>>> http://www.secmotic.com
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
>>> puede contener información privilegiada o confidencial y es para uso
>>> exclusivo de la persona o entidad de destino. Si no es usted. el
>>> destinatario indicado, queda notificado de que la lectura, utilización,
>>> divulgación y/o copia sin autorización puede estar prohibida en virtud de
>>> la legislación vigente. Si ha recibido este mensaje por error, le rogamos
>>> que nos lo comunique inmediatamente por esta misma vía y proceda a su
>>> destrucción.
>>>
>>> The information contained in this transmission is privileged and
>>> confidential information intended only for the use of the individual or
>>> entity named above. If the reader of this message is not the intended
>>> recipient, you are hereby notified that any dissemination, distribution or
>>> copying of this communication is strictly prohibited. If you have received
>>> this transmission in error, do not read it. Please immediately reply to the
>>> sender that you have received this communication in error and then delete
>>> it.
>>>
>>> Esta mensagem e seus anexos se dirigem exclusivamente ao seu
>>> destinatário, pode conter informação privilegiada ou confidencial e é para
>>> uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o
>>> destinatário indicado, fica notificado de que a leitura, utilização,
>>> divulgação e/ou cópia sem autorização pode estar proibida em virtude da
>>> legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos
>>> o comunique imediatamente por esta mesma via e proceda a sua destruição
>>>
>>
>>
>>
>> --
>> Carlos Corrales
>> Cofounder Secmotic Innovation S.L.
>> http://www.secmotic.com
>>
>>
>
>
> --
> Carlos Corrales
> Cofounder Secmotic Innovation S.L.
> http://www.secmotic.com
>
>


-- 
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-finodex-coaching/attachments/20150914/ef939ea0/attachment.html>


More information about the Fiware-finodex-coaching mailing list

You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy   Cookies policy