[Fiware-lab-help] Hive - JSonSerde library - Cygnus

[Paparella Francesco]

Hi ,
I have configured the Cygnus instance with the following configuration for Hive:

cygnus-ngsi.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org
cygnus-ngsi.sinks.hdfs-sink.hive.server_version = 2
cygnus-ngsi.sinks.hdfs-sink.file_format = json-column
cygnus-ngsi.sinks.hdfs-sink.csv_separator = ,
cygnus-ngsi.sinks.hdfs-sink.hdfs_host = storage.cosmos.lab.fiware.org
cygnus-ngsi.sinks.hdfs-sink.hdfs_port = 14000
cygnus-ngsi.sinks.hdfs-sink.hdfs_username = <username>
cygnus-ngsi.sinks.hdfs-sink.hdfs_password = <password>
cygnus-ngsi.sinks.hdfs-sink.oauth2_token = <token>
cygnus-ngsi.sinks.hdfs-sink.hive = true


After table creation, I have received the below error message:


Error message:
SQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
SQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
  Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
  Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe

Ddl hive table:
create external table if not exists default.mellusom_defaultx002fx0053x0043x0053x002d004_x0053martx0043ityx0053ensor_column
(
recvx0054ime string,
fiwarex0053ervicex0050ath string,
entityx0049d string,
entityx0054ype string,
brightness string,
brightness_md array<struct<name:string,type:string,value:string>>,
city string,city_md array<struct<name:string,type:string,value:string>>,
continent string,continent_md array<struct<name:string,type:string,value:string>>,
country string,country_md array<struct<name:string,type:string,value:string>>,
location string,location_md array<struct<name:string,type:string,value:string>>,
owner string,owner_md array<struct<name:string,type:string,value:string>>,
region string,region_md array<struct<name:string,type:string,value:string>>,
temperature_c string,temperature_c_md array<struct<name:string,type:string,value:string>>,
zip string,zip_md array<struct<name:string,type:string,value:string>>)
row format serde 'org.openx.data.jsonserde.JsonSerDe' with serdeproperties ("dots.in.keys" = "true")
location '/user/mellusom/default/SCS-004_SmartCitySensor'


I solved the issue regarding the error message using this link:

https://community.cloudera.com/t5/Batch-SQL-Apache-Hive/Where-to-put-json-serde-1-3-7-jar-with-dependencies-jar-file/td-p/48073<http://cp.mcafee.com/d/avndygsd3gOrhvKYY-U-NsTsSyyPtBBBVBMSyyOyevKUYyrhhp76zBWXVEVd79J55VNdNZcQszzrItxrWmHOSHr7PPZ1lGNcWuvaDiCrJm9DjPVkWkPuE9LK-qen-LOrz1EVd7bPTnKnjpujVxUsUOUVR4mlHFGTppVkffGhBrwqrhdI6XYCyOzsQsIL8I6XCOsVHkiP5RmUiCmWNZmQfzLbVd_1DpS9BNw1MGn45JIjYK5V6vM1ivNYLeFtpKpshFm1uDNgYKynpbzyUzIiVykOpt3RLwxYbi-xshS9tSh_Vd0KJz3RNP9IhHjV4orqluJIo_fUR1jIz6N_w09JAsYYyCr1vF6y0bOc-q82VEw0woH0o-a7Cy0-h2eNb6y0Kq818IhwTgSOedBjl2gIAwSd_>


and after that I have tried to deploy on my local Hive istance the jar library recovered by http://www.congiu.net/hive-json-serde/<http://cp.mcafee.com/d/FZsS93gwrhvKYY-U-NsTsSyyPtBBBVBMSyyOyevKUYyrhhp76zBWXVEVd79J55VNdNZcQszzrItxrWmHOSHr7PPZ1lGNcWuvaDiCrJm9DjPVkWkPuE9LK-qen-LOrz1EVd7bPTnKnjpujVxUsUOUVR4mlHFGTppVkffGhBrwqrjdI6XYCyOzsQsIL8I6XCM0lGM8OrU6UjApYK8QH0LjUEvxIhHjV4orqluJIo_fUR1jIz6N_w09JAsYYyCr1vF6y0bOc-q82VEw0woH0o-a7Cy0-h2eNb6y0Kq818IhwTgSOedFPnyNnzG> (version 1.3.8)
and now It's correctly working in local.


Best regards.



Francesco Paparella
Ricerca e Sviluppo

Corvallis S.p.A. a Socio Unico
Attività di direzione e coordinamento Corvallis Holding SpA
Gruppo Corvallis
Viale Aldo Moro, 36
73100 Lecce
Italia

Telefono         +39 0832 069901
Telefax   +39 0832 069911

francesco.paparella at corvallis.it<mailto:francesco.paparella at corvallis.it>

www.corvallis.it<http://www.corvallis.it/>

Le informazioni contenute in questo messaggio di posta elettronica e negli eventuali allegati sono riservate e confidenziali e sono indirizzate esclusivamente al destinatario. Si prega di non fare copia, inoltrare a terzi o conservare tale messaggio se non si è il legittimo destinatario dello stesso. Qualora questo messaggio sia stato ricevuto per errore, si prega di rinviarlo al mittente e di cancellarlo permanentemente dal proprio computer.

The information contained in this message and in any attachment is intended exclusively for the recipient. If you are not the intended recipient you are hereby notified not to copy, save, disclose, or distribute it to any third party. If you erroneously received this message you are kindly requested  to return it to the sender and eliminate it permanently from your computer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-lab-help/attachments/20171024/6f7e9f6f/attachment-0001.html>