[Fiware-lab-help] Automatic creation of the hive tables on cosmos

FRANCISCO ROMERO BUENO francisco.romerobueno at telefonica.com
Sun Feb 22 14:43:39 CET 2015

Previous message: [Fiware-lab-help] Automatic creation of the hive tables on cosmos
Next message: [Fiware-lab-help] [FI-WARE-JIRA] (FIL-38) How to instantiate a VM from a recipe ?
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hi Antonio,

Yes, I know. Even, I added a comment 2 days ago.

I’ll close the issue.

Regards,
Francisco

De: antonio caristia <a.caristia at gmail.com<mailto:a.caristia at gmail.com>>
Fecha: domingo, 22 de febrero de 2015, 11:48
Para: Francisco Romero Bueno <francisco.romerobueno at telefonica.com<mailto:francisco.romerobueno at telefonica.com>>
CC: "gioakbombaci at gmail.com<mailto:gioakbombaci at gmail.com>" <gioakbombaci at gmail.com<mailto:gioakbombaci at gmail.com>>, "fiware-lab-help at lists.fi-ware.org<mailto:fiware-lab-help at lists.fi-ware.org>" <fiware-lab-help at lists.fi-ware.org<mailto:fiware-lab-help at lists.fi-ware.org>>
Asunto: Re: Automatic creation of the hive tables on cosmos

Hi Francisco,
sorry for the delay in the response but in these days i'm not going to work to finish my thesis, I graduate in March.
I have received an answer on StackOverflow about the problem of the hive tables in Cygnus. This is the link of the response:
http://stackoverflow.com/questions/27918545/automatic-creation-of-the-hive-tables-on-cosmos?noredirect=1#comment45506173_27918545
It should be useful for you.
Kind regagrds,

Antonio

2015-02-06 8:57 GMT+01:00 FRANCISCO ROMERO BUENO <francisco.romerobueno at telefonica.com<mailto:francisco.romerobueno at telefonica.com>>:
Hi Antonio,

Could you perform the following test? Start Cygnus as always, and then create a new fresh entity at Orion and a subscription regarding this new entity. Finally, send me the entire log, please. The best log would be that created in debug mode, so start Cygnus with -Dflume.root.logger=DEBUG,LOGFILE

Thanks!
Francisco

De: antonio caristia [mailto:a.caristia at gmail.com<mailto:a.caristia at gmail.com>]
Enviado el: jueves, 29 de enero de 2015 17:50

Para: FRANCISCO ROMERO BUENO
Asunto: Re: Automatic creation of the hive tables on cosmos

Hi Francisco,
I have used a server in python that receives the subscriptions from orion and creates the tables
using hive api.
At this moment i use this workaround, problably i'm not able to configure correctly cygnus.
If you have some ideas to help me, you can send me an email to this address :)
Thanks for your reply.

Best regards,

Antonio

2015-01-29 15:43 GMT+01:00 FRANCISCO ROMERO BUENO <francisco.romerobueno at telefonica.com<mailto:francisco.romerobueno at telefonica.com>>:
Hi Antonio,

Sorry for the delay in continuing the conversation.

I’ve seen that these tables, associated to your username, have been created:

gioakbombaci_pm10_142245780725_observations
gioakbombaci_pm10_142245909633_observations
gioakbombaci_pm10_142245960442_observations
gioakbombaci_pm10_142246020579_observations
gioakbombaci_pm10_142246134747_observations
gioakbombaci_pm10_142246257881_observations
gioakbombaci_pm10_142246363018_observations
gioakbombaci_pm10_142246475644_observations
gioakbombaci_pm10_14224658677_observations
gioakbombaci_pm10_142246691926_observations
gioakbombaci_pm10_14224683607_observations
gioakbombaci_pressure_142245780725_observations
gioakbombaci_pressure_142245909633_observations
gioakbombaci_pressure_142245960442_observations
gioakbombaci_pressure_142246020579_observations
gioakbombaci_pressure_142246134747_observations
gioakbombaci_pressure_142246257881_observations
gioakbombaci_pressure_142246363018_observations
gioakbombaci_pressure_142246475644_observations
gioakbombaci_pressure_14224658677_observations
gioakbombaci_pressure_142246691926_observations
gioakbombaci_pressure_14224683607_observations
gioakbombaci_temperature_142245780725_observations
gioakbombaci_temperature_142245909633_observations
gioakbombaci_temperature_142245960442_observations
gioakbombaci_temperature_142246020579_observations
gioakbombaci_temperature_142246134747_observations
gioakbombaci_temperature_142246257881_observations
gioakbombaci_temperature_142246363018_observations
gioakbombaci_temperature_142246475644_observations
gioakbombaci_temperature_14224658677_observations
gioakbombaci_temperature_142246691926_observations
gioakbombaci_temperature_14224683607_observations

Did you created them manually or were they created by Cygnus?

Regards,
Francisco

De: antonio caristia [mailto:a.caristia at gmail.com<mailto:a.caristia at gmail.com>]
Enviado el: lunes, 19 de enero de 2015 11:13

Para: FRANCISCO ROMERO BUENO
Asunto: Re: Automatic creation of the hive tables on cosmos

Hi Francisco,

have you any news for my issue?
If it is not possible to resolve at this moment, can you give me some info how to create
the tables manually or with bash/python scripts?

I can't test my widgets without data to query.

Thanks in advance, best regards.

Antonio

2015-01-15 16:57 GMT+01:00 antonio caristia <a.caristia at gmail.com<mailto:a.caristia at gmail.com>>:
Hi Francisco,

I have modified cygnus.conf and created matching_table.conf on [MY_CYGNUS_HOME]/conf.
I have no more the error in flume.log but cygnys does not create the tables yet.
I have attached my flume.log and cygnus.conf.

Regards,

Antonio

2015-01-15 12:08 GMT+01:00 FRANCISCO ROMERO BUENO <francisco.romerobueno at telefonica.com<mailto:francisco.romerobueno at telefonica.com>>:
Hi Antonio,

A couple of questions:

· Where do you want to persist your context data? Only Cosmos/HDFS for Hive querying? Because you have also activated the MySQL and CKAN sinks, which is leading to lot of errors due these sinks are not properly configured. Just edit these lines:

o cygnusagent.sinks = hdfs-sink # i.e. remove mysql-sink ckan-sink

o cygnusagent.channels = hdfs-channel @ i.e. remove mysql-channel ckan-channel

o The OrionMySQLSink, OrionCKANSink, ckan-channel and mysql-channel configuration parts may be removed as well.

· Do you have such /usr/cygnus/conf/matching_table.conf file? This is the default location for the destination extractor matching rules, when installing through the RPM. How did you installed Cygnus? If you installed from sources, just create an empty matching_table.conf file under [YOUR_CYGNUS_HOME]/conf folder.

I’m not sure if these issues are related or not to the Hive table problem, but fixing them will lead to a more clean log file :)

Regards,
Francisco

De: antonio caristia [mailto:a.caristia at gmail.com<mailto:a.caristia at gmail.com>]
Enviado el: jueves, 15 de enero de 2015 10:50

Para: FRANCISCO ROMERO BUENO
Asunto: Re: Automatic creation of the hive tables on cosmos

Hi Francisco,

I do not use any matching rules, in the log file i have this no such file error.

[root at orionold logs]# cat flume.log |grep table
09 Jan 2015 10:30:09,306 ERROR [lifecycleSupervisor-1-2] (es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor.initialize:84) - Runtime error (File not found. Details=/usr/cygnus/conf/matching_table.conf (No such file or directory))
09 Jan 2015 11:04:15,009 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (es.tid.fiware.fiwareconnectors.cygnus.backends.hdfs.HDFSBackend.provisionHiveTable:135) - Creating Hive external table=gioakbombaci_def_serv_Taxi1_Taxi_column
13 Jan 2015 12:33:42,158 ERROR [lifecycleSupervisor-1-3] (es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor.initialize:84) - Runtime error (File not found. Details=/usr/cygnus/conf/matching_table.conf (No such file or directory))
13 Jan 2015 12:36:18,190 ERROR [lifecycleSupervisor-1-2] (es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor.initialize:84) - Runtime error (File not found. Details=/usr/cygnus/conf/matching_table.conf (No such file or directory))
13 Jan 2015 12:57:12,137 ERROR [lifecycleSupervisor-1-2] (es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor.initialize:84) - Runtime error (File not found. Details=/usr/cygnus/conf/matching_table.conf (No such file or directory))

This is the url https://drive.google.com/file/d/0B5QWQPdKi7qvbzE5aVkyZi03NUU/view?usp=sharing on my google drive where you can find flume.log

The Entity is Taxi1, the type is Taxi.
These are the attributes:

1. name = temperature, type = centigrade
2. name = pressure, type = mmHg
3. name = PM10, type = concentration
4. name = taxiPosition, type = coords
5. name = Service1, type = text
Thanks for your help, best regards.

Antonio

2015-01-15 9:27 GMT+01:00 FRANCISCO ROMERO BUENO <francisco.romerobueno at telefonica.com<mailto:francisco.romerobueno at telefonica.com>>:
Hi Antonio,

The configuration seems to be OK. Can you send me the related logs?

In addition, which is the name and type of the NGIS entity whose data is being persisted through Cygnus? Are you using any matching rule at the destination extractor matching table?

Regards,
Francisco

De: antonio caristia [mailto:a.caristia at gmail.com<mailto:a.caristia at gmail.com>]
Enviado el: martes, 13 de enero de 2015 16:21
Para: FRANCISCO ROMERO BUENO
Asunto: Re: Automatic creation of the hive tables on cosmos

Hi Francisco,
thanks for your reply.
I use cygnus 0.6, i have attached my cygnus.conf.

Thanks for your help, best regards.

Antonio

2015-01-13 13:13 GMT+01:00 FRANCISCO ROMERO BUENO <francisco.romerobueno at telefonica.com<mailto:francisco.romerobueno at telefonica.com>>:
Hi Antonio,

Sorry for the delay, but I was on vacations :)

Can you send me the Cygnus configuration you are using? Which Cygnus version indeed?

Regards,
Francisco

De: antonio caristia [mailto:a.caristia at gmail.com<mailto:a.caristia at gmail.com>]
Enviado el: viernes, 09 de enero de 2015 12:08
Para: FRANCISCO ROMERO BUENO
Asunto: Automatic creation of the hive tables on cosmos

Hi Francisco,
I have an Orion context broker and i use cygnus to send data on my cosmos account.
Cygnus create correctly the txt files with the data on my cosmos account, but it does not create
the tables for hive. I would query my data through an hive client.
Is there a configuration paramater to set in cygnus.conf to do that?
How can i solve?

Thanks in advance.
Best regards,

Antonio

________________________________

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

________________________________

Previous message: [Fiware-lab-help] Automatic creation of the hive tables on cosmos
Next message: [Fiware-lab-help] [FI-WARE-JIRA] (FIL-38) How to instantiate a VM from a recipe ?
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the Fiware-lab-help mailing list

You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy