Hi Francisco,
Yes, we were using cygnus 0.5-91... It is the default version when using
the image available in FIWARE lab (orion-psb-image-R4.2
<https://cloud.lab.fiware.org/#nova/images/7dd7a0cc-3e3b-4eb3-a50c-4a2e3e498df3>
).
After updating the version and configuring it properly, everything is
working again.
Thank you very much!
2015-09-15 9:45 GMT+02:00 MANUEL ESCRICHE VICENTE <
manuel.escrichevicente at telefonica.com>:
> Be aware there exist this issue in JIRA for this exchange:
> https://jira.fiware.org/browse/HELC-1025
>
>
>
>
>
> *From:* FRANCISCO ROMERO BUENO
> *Sent:* martes, 15 de septiembre de 2015 9:12
> *To:* Carlos Corrales; MANUEL ESCRICHE VICENTE
> *Cc:* JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching at lists.fi-ware.org;
> jose at secmotic.com
>
> *Subject:* Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
>
>
>
> Hi Carlos,
>
>
>
> I will aggregate through this email all the other emails you sent.
>
>
>
> Regarding the version, it is printed in the log at the very beginning of
> the execution. In addition, if you installed via RPM you can check for the
> installed version by typing:
>
>
>
> $ rpm –qa | grep cygnus
>
>
>
> Nevertheless, looking at your configuration, it seems you are using a very
> old version of Cygnus, or you are mixing parameters from different
> versions. I mean:
>
> - Current version is 0.8.2
> - The "cygnusagent.sinks.hdfs-sink.type =
> es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink” part is previous
> to 0.8.0. From 0.8.0 it must be
> “com.telefonica.iot.cygnus.sinks.OrionCKANSink”
> - The OAuth2 token was introduced in 0.8.2
>
> My recommendation is you install the latest version from RPM (
> https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1) and
> you configure Cygnus following the guidelines appearing in the master or
> release-0.8.2 branches (they are synchronized):
>
> https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md
>
> https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md
> <https://github.com/telefonicaid/fiware-cygnus/tree/release/0.8.2>
>
>
>
> Regarding the fiware-service and fiware-servicePath values, if they are
> not notified as http headers, then the default configured values in Cygnus’
> HTTPSource are used. Again, it seems you are not using/configuring these
> values.
>
>
>
> Regards,
>
> Francisco
>
>
>
> *De: *Carlos Corrales <carlos at secmotic.com>
> *Fecha: *lunes, 14 de septiembre de 2015, 18:22
> *Para: *MANUEL ESCRICHE VICENTE <manuel.escrichevicente at telefonica.com>
> *CC: *JUAN JOSE HIERRO SUREDA <juanjose.hierro at telefonica.com>, "
> fiware-finodex-coaching at lists.fi-ware.org" <
> fiware-finodex-coaching at lists.fi-ware.org>, "jose at secmotic.com" <
> jose at secmotic.com>, Francisco Romero Bueno <
> francisco.romerobueno at telefonica.com>
> *Asunto: *Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
>
>
>
> Hi everyone,
>
> Thanks for your responses, we have been testing the whole day and trying
> to find out the issue. It is not a problem with neither the password, nor
> the oauth2 token. We are able to access using ssh user/password and the
> curl command proposed by Pasquale with Oauth2 token. This is what I
> received using the Oauth2 token curl proposed by Pasquale
>
> curl -X GET "
> http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado"
> -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
>
> {"FileStatuses":{"FileStatus":[{"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0},{"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0},{"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0},{"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0},{"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0},{"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}]}}
>
> Everything seems to be correct. However, using the same token and updating
> ORION, I receive the following error in Cygnus:
>
> time=2015-09-14T18:10:44.419CEST | lvl=ERROR |
> trans=1442246992-282-0000000000 | function=process | comp=Cygnus |
> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] :
> Persistence error (The org42/idelgado_tablanull directory could not be
> created in HDFS. HttpFS response: 401 Unauthorized)
> time=2015-09-14T18:10:44.419CEST | lvl=WARN |
> trans=1442246992-282-0000000000 | function=process | comp=Cygnus |
> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event
> TTL has expired, it is no more re-injected in the channel (id=1444248959,
> ttl=0)
>
> However, if I try to access the directory shown in the error, I'm able to
> do so
>
> ssh idelgado at cosmos.lab.fi-ware.org
> hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
> Found 1 items
> -rw-r----- 3 idelgado idelgado 1416 2015-07-01 13:11
> /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt
>
> My cygnus.conf file content is:
>
> #
> ============================================
>
> # OrionHDFSSink
> configuration
>
> # channel name from where to read notification
> events
> cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
> # sink class, must not be
> changed
>
> cygnusagent.sinks.hdfs-sink.type =
> es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
> # Comma-separated list of FQDN/IP address regarding the Cosmos Namenode
> endpoints
> cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
> # port of the Cosmos service listening for persistence operations; 14000
> for httpfs, 50070 for webhdfs and free c\
> hoice for
> inifinty
>
> cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
> # default username allowed to write in
> HDFS
> cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado
> # default password for the default
> username
>
> #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
>
> cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i
> # HDFS backend type (webhdfs, httpfs or
> infinity)
> cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
> # how the attributes are stored, either per row either per column (row,
> column)
> cygnusagent.sinks.hdfs-sink.attr_persistence = column
> # prefix for the database and table names, empty if no prefix is
> desired
> cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla
> # Hive FQDN/IP address of the Hive
> server
>
> cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
> # Hive port for Hive external table
> provisioning
>
> cygnusagent.sinks.hdfs-sink.hive_port = 10000
>
> Could you please help us?
>
> Thanks in advance!
>
>
>
>
>
>
>
> 2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <
> manuel.escrichevicente at telefonica.com>:
>
> In this case I see both Pasquale (Coach) and Francisco (GE Owner) have
> reacted to the request.
>
> In the normal flow GE owners enter into action upon request from Coaches.
>
> Kind regards,
>
> Manuel
>
>
>
> *From:* JUAN JOSE HIERRO SUREDA
> *Sent:* lunes, 14 de septiembre de 2015 0:37
> *To:* Carlos Corrales
> *Cc:* fiware-finodex-coaching at lists.fi-ware.org; jose at secmotic.com;
> FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
> *Subject:* Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
>
>
>
>
> I guess that the FINODEX coach will forward the ticket to the FIWARE GE
> owner (or eventually, if he hadn't know where to redirect it, he would
> redirect the original question to the fiware-tech-help mailing list)
>
> Point is that one way or the other, it would arrive at the inbox of the
> FIWARE GE owner who has to take care of the ticket.
>
> You may have sent the ticket directly to the fiware-tech-help mailing
> list ... however, I guess that the FINODEX coach team wanted to be informed
> to keep track of the evolution.
>
> Manuel, in copy, is the master of all the helpdesk processes and he can
> clarify what the actual workflow is.
>
> Best regards,
>
> -- Juanjo
>
> ______________________________________________________
>
>
>
> Coordinator and Chief Architect, FIWARE platform
>
> CTO Industrial IoT, Telefónica
>
>
>
> email: juanjose.hierro at telefonica.com
>
> twitter: @JuanjoHierro
>
>
>
> You can follow FIWARE at:
>
> website: http://www.fiware.org
>
> twitter: @FIWARE
>
> facebook: http://www.facebook.com/pages/FI-WARE/251366491587242
>
> linkedIn: http://www.linkedin.com/groups/FIWARE-4239932
>
> On 13/09/15 16:36, Carlos Corrales wrote:
>
> Hi Juanjo,
>
> Thanks for you feedback, we sent it today because we wanted you to answer
> it tomorrow morning :).
>
>
>
> As you can see, a ticket has been created in Jira (
> https://jira.fiware.org/browse/HELC-1025). I used the
> fiware-finodex-coaching list, because we are a FINODEX project and they
> told us to do so. Should we create another ticket in the fiware-tech-help
> list also?
>
> Best regards
>
>
>
> 2015-09-13 16:30 GMT+02:00 Juanjo Hierro <juanjose.hierro at telefonica.com>:
>
> Hi,
>
> I copy Francisco (Paco) Romero who should be able to provide some help
> on this. I guess he won't be able to work on the matter until tomorrow
> since support is not granted out of office working hours.
>
> Nevertheless, I believe that it is rather important that you open an
> issue so that a ticket on our help-desk system is created and then
> follow-up of support is feasible.
>
> You may create a ticket in our helpdesk simply by sending an email to
> either fiware-lab-help (if it happens to be an issue which is mostly
> related to FIWARE Lab or the global COSMOS instance setup on the FIWARE
> Lab) or fiware-tech-help (if it happens to be an issue mostly related with
> the technology, i.e., with COSMOS in general and not the particular
> instance of COSMOS running on the FIWARE Lab).
>
> Best regards,
>
> -- Juanjo
>
> ______________________________________________________
>
>
>
> Coordinator and Chief Architect, FIWARE platform
>
> CTO Industrial IoT, Telefónica
>
>
>
> email: juanjose.hierro at telefonica.com
>
> twitter: @JuanjoHierro
>
>
>
> You can follow FIWARE at:
>
> website: http://www.fiware.org
>
> twitter: @FIWARE
>
> facebook: http://www.facebook.com/pages/FI-WARE/251366491587242
>
> linkedIn: http://www.linkedin.com/groups/FIWARE-4239932
>
> On 13/09/15 15:20, Carlos Corrales wrote:
>
> Hi everyone,
>
> I'm Carlos Corrales, part of the development team at SAPIN (FINODEX012).
> We have been working hard in the application and now that it's finished and
> we are doing tests with final users, we are improving our code. A couple of
> months ago we were able to write data to our HDFS in the COSMOS Fiware
> machine (cosmos.lab.fi-ware.org) with our user (idelgado). As we are not
> using Cosmos for production right now (it will be used in the future), we
> left it aside. Now that we are checking that everything works as expected,
> we can see that our Cygnus connector is not able to write properly in HDFS.
> We haven't changed anything in the configuration, but we receive the
> following error when trying to write (flume.log file)
>
> time=2015-09-13T15:11:36.744CEST | lvl=INFO |
> trans=1442149853-658-0000000000 | function=persist | comp=Cygnus |
> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink[367] :
> [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
> (idelgado_SAPINnull), Data
> ({"recvTime":"2015-09-13T15:11:00.413","age":"25",
> "age_md":[],"exercisetype":"Light", "exercisetype_md":[],"hr":"36",
> "hr_md":[],"id_session":"1234", "id_session_md":[]})
> time=2015-09-13T15:11:36.803CEST | lvl=ERROR |
> trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] :
> Persistence error (The unknownt/idelgado_SAPINnull directory could not be
> created in HDFS. HttpFS response: 401 Unauthorized)
> time=2015-09-13T15:11:36.804CEST | lvl=INFO |
> trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[150] : An event
> was put again in the channel (id=1444248959, ttl=0)
> time=2015-09-13T15:11:36.804CEST | lvl=INFO |
> trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
> msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[175] : Finishing
> transaction (1442149853-658-0000000000)
>
> As you can see, the error is 401 Unauthorized. However, we can access the
> Cosmos machine using SSH, and the user/password in the cygnus.conf file. We
> also checked that the IP address of the Cosmos machine hasn't changed.
>
> Could you please help us? We don't know what else to check...
>
> Best regards
>
>
> --
>
> Carlos Corrales
>
> Cofounder Secmotic Innovation S.L.
>
> http://www.secmotic.com
>
>
>
> _______________________________________________
>
> Fiware-finodex-coaching mailing list
>
> Fiware-finodex-coaching at lists.fiware.org
>
> https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
>
>
> ------------------------------
>
>
> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
> puede contener información privilegiada o confidencial y es para uso
> exclusivo de la persona o entidad de destino. Si no es usted. el
> destinatario indicado, queda notificado de que la lectura, utilización,
> divulgación y/o copia sin autorización puede estar prohibida en virtud de
> la legislación vigente. Si ha recibido este mensaje por error, le rogamos
> que nos lo comunique inmediatamente por esta misma vía y proceda a su
> destrucción.
>
> The information contained in this transmission is privileged and
> confidential information intended only for the use of the individual or
> entity named above. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this transmission in error, do not read it. Please immediately reply to the
> sender that you have received this communication in error and then delete
> it.
>
> Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário,
> pode conter informação privilegiada ou confidencial e é para uso exclusivo
> da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário
> indicado, fica notificado de que a leitura, utilização, divulgação e/ou
> cópia sem autorização pode estar proibida em virtude da legislação vigente.
> Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique
> imediatamente por esta mesma via e proceda a sua destruição
>
>
>
>
> --
>
> Carlos Corrales
>
> Cofounder Secmotic Innovation S.L.
>
> http://www.secmotic.com
>
>
>
>
> ------------------------------
>
>
> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
> puede contener información privilegiada o confidencial y es para uso
> exclusivo de la persona o entidad de destino. Si no es usted. el
> destinatario indicado, queda notificado de que la lectura, utilización,
> divulgación y/o copia sin autorización puede estar prohibida en virtud de
> la legislación vigente. Si ha recibido este mensaje por error, le rogamos
> que nos lo comunique inmediatamente por esta misma vía y proceda a su
> destrucción.
>
> The information contained in this transmission is privileged and
> confidential information intended only for the use of the individual or
> entity named above. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this transmission in error, do not read it. Please immediately reply to the
> sender that you have received this communication in error and then delete
> it.
>
> Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário,
> pode conter informação privilegiada ou confidencial e é para uso exclusivo
> da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário
> indicado, fica notificado de que a leitura, utilização, divulgação e/ou
> cópia sem autorização pode estar proibida em virtude da legislação vigente.
> Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique
> imediatamente por esta mesma via e proceda a sua destruição
>
>
>
>
> --
>
> Carlos Corrales
>
> Cofounder Secmotic Innovation S.L.
>
> http://www.secmotic.com
>
> ------------------------------
>
> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
> puede contener información privilegiada o confidencial y es para uso
> exclusivo de la persona o entidad de destino. Si no es usted. el
> destinatario indicado, queda notificado de que la lectura, utilización,
> divulgación y/o copia sin autorización puede estar prohibida en virtud de
> la legislación vigente. Si ha recibido este mensaje por error, le rogamos
> que nos lo comunique inmediatamente por esta misma vía y proceda a su
> destrucción.
>
> The information contained in this transmission is privileged and
> confidential information intended only for the use of the individual or
> entity named above. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this transmission in error, do not read it. Please immediately reply to the
> sender that you have received this communication in error and then delete
> it.
>
> Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário,
> pode conter informação privilegiada ou confidencial e é para uso exclusivo
> da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário
> indicado, fica notificado de que a leitura, utilização, divulgação e/ou
> cópia sem autorização pode estar proibida em virtude da legislação vigente.
> Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique
> imediatamente por esta mesma via e proceda a sua destruição
>
--
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-finodex-coaching/attachments/20150916/ac805022/attachment.html>
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy