[Fiware-incense-coaching] Heaving problems perssiting data from Cygnus to Cosmos global instanse

[Menachem Shtalhaim]

I have looked at the log and found the following:
10 Nov 2015 16:39:57,903 INFO
 [SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistAggregation:918)  -
[hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
(def_serv/def_servpath/8_registervalues/8_registervalues.txt), Data
({"recvTime":"2015-11-10T15:39:39.919Z","fiwareservicepath":"def_servpath","entityId":"8","entityType":"RegisterValues",
"value":"1959", "value_md":[], "name":"register number 8", "name_md":[]})
10 Nov 2015 16:39:58,183 ERROR
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:305)  - Persistence
error (The /user/ms/def_serv/def_servpath/8_registervalues directory could
not be created in HDFS. Server response: 503 Service unavailable)
10 Nov 2015 16:39:58,183 INFO
 [SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:307)  - Finishing
transaction (1447169873-477-0000000000)



Is there anything wrong with the configuration I am using?


*Agent_1.conf:*

# Copyright 2014 Telefֳ³nica Investigaciֳ³n y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License
along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License
please contact with iot_support at tid dot es

#=============================================
# To be put in APACHE_FLUME_HOME/conf/agent.conf
#
# General configuration template explaining how to setup a sink of each of
the available types (HDFS, CKAN, MySQL).

#=============================================
# The next tree fields set the sources, sinks and channels used by Cygnus.
You could use different names than the
# ones suggested below, but in that case make sure you keep coherence in
properties names along the configuration file.
# Regarding sinks, you can use multiple types at the same time; the only
requirement is to provide a channel for each
# one of them (this example shows how to configure 3 sink types at the same
time). Even, you can define more than one
# sink of the same type and sharing the channel in order to improve the
performance (this is like having
# multi-threading).
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel

#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel
# source class, must not be changed
cygnusagent.sources.http-source.type =
org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming
notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler =
com.telefonica.iot.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = def_serv
# Default service path (service path semantic depends on the persistence
sink)
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
# Number of channel re-injection retries before a Flume event is definitely
discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type =
com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path
to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file =
/usr/cygnus/conf/grouping_rules.conf

# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type =
com.telefonica.iot.cygnus.sinks.OrionHDFSSink
#ֲ true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.hdfs-sink.enable_grouping = false
# rest if the interaction with HDFS will be WebHDFS/HttpFS-based, binary if
based on the Hadoop API
cygnusagent.sinks.hdfs-sink.backend_impl = rest
# Comma-separated list of FQDN/IP address regarding the HDFS Namenode
endpoints
# If you are using Kerberos authentication, then the usage of FQDNs instead
of IP addresses is mandatory
cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fiware.org
# port of the HDFS service listening for persistence operations; 14000 for
httpfs, 50070 for webhdfs
cygnusagent.sinks.hdfs-sink.hdfs_port = 14000
# username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.hdfs_username = COSMOS_USER
#ֲ password for the above username; this is only required for Hive
authentication
cygnusagent.sinks.hdfs-sink.hdfs_password = COSMOS_PASSWORD
# OAuth2 token for HDFS authentication
#cygnusagent.sinks.hdfs-sink.oauth2_token = TOKEN
# how the attributes are stored, available formats are json-row,
json-column, csv-row and csv-column
cygnusagent.sinks.hdfs-sink.file_format = json-column
#ֲ number of notifications to be included within a processing batch
cygnusagent.sinks.hdfs-sink.batch_size = 100
# timeout for batch accumulation
cygunsagent.sinks.hdfs-sink.batch_timeout = 30
# Hive enabling
cygnusagent.sinks.hdfs-sink.hive = true
# Hive server version, 1 or 2 (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.server_version = 2
# Hive FQDN/IP address of the Hive server (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org
# Hive port for Hive external table provisioning (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.port = 10000
# Kerberos-based authentication enabling
cygnusagent.sinks.hdfs-sink.krb5_auth = false
# Kerberos username (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
# Kerberos password (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
# Kerberos login file (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file =
/usr/cygnus/conf/krb5_login.conf
# Kerberos configuration file (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file =
/usr/cygnus/conf/krb5.conf


#=============================================
# hdfs-channel configuration
# channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory
# capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100





*CYGNUS_INSTANCE_1.CONF*


#####
#
# Configuration file for apache-flume
#
#####
# Copyright 2014 Telefonica Investigaciֳ³n y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License
along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License
please contact with iot_support at tid dot es

# Who to run cygnus as. Note that you may need to use root if you want
# to run cygnus in a privileged port (<1024)
CYGNUS_USER=cygnus

# Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf

# Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_1.conf

# Name of the agent. The name of the agent is not trivial, since it is the
base for the Flume parameters
# naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnusagent

# Name of the logfile located at /var/log/cygnus. It is important to put
the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log

# Administration port. Must be unique per instance
ADMIN_PORT=8081

# Polling interval (seconds) for the configuration reloading
POLLING_INTERVAL=30




Thanks you!

-- 

*Menachem Shtalhaim*

menachem.shtalhaim at n-join.com <elkana at n-join.com> | +972-5
<%2B972-527-213941>47241959 | www.n-join.com

*n-Join – Maximizing industrial potential*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-incense-coaching/attachments/20151110/4b6d91e3/attachment.html>