When trying to persist an entity from Cygnus to Cosmos global instance it
fails.
Looking at the log file I see something like that:
"2015 15:31:50,006 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:273) -
>> GET
/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&
user.name=ms HTTP/1.1 12 Nov 2015 15:31:50,006 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> X-Auth-Token: xxxxxxxxxxxxxxxxxxx 12 Nov 2015 15:31:50,006 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> Host: cosmos.lab.fiware.org:14000 12 Nov 2015 15:31:50,006 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> Connection: Keep-Alive 12 Nov 2015 15:31:50,007 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> User-Agent: Apache-HttpClient/4.2.1 (java 1.5) 12 Nov 2015 15:31:50,284
DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.close:169) - Connection
0.0.0.0:59834<->130.206.80.46:14000 closed 12 Nov 2015 15:31:50,285 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.client.DefaultRequestDirector.tryExecute:716) -
Closing the connection. 12 Nov 2015 15:31:50,285 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.close:169) - Connection
0.0.0.0:59834<->130.206.80.46:14000 closed 12 Nov 2015 15:31:50,286 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.client.DefaultRequestDirector.tryExecute:723) - I/O
exception (java.net.SocketException) caught when processing request:
Connection reset 12 Nov 2015 15:31:50,287 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.client.DefaultRequestDirector.tryExecute:728) -
Connection reset java.net.SocketException: Connection reset at
java.net.SocketInputStream.read(SocketInputStream.java:185) at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
"
(I have obfuscated the token in the example above.)
for some reason the connection is closed.
Orion and Cygnus are installed on a fiware lab private instance in Spain2.
When trying to fire the same request from the command line from the Orion
machine, I get the same disconnected response.
curl -X GET "
http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&user.name=ms"
-H "X-Auth-Token: xxxxxxxxxxxxxxxxxxxxxx" curl: (56) Failure when receiving
data from the peer
But surprisingly, when doing the same request from the command line from
the Cosmos global instance, it works!
See:
[ms at cosmosmaster-gi ~]$ curl -X GET "
http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&user.name=ms"
-H "X-Auth-Token: xxxxxxxxxxxxxxxxx"
{"FileStatus":{"pathSuffix":"","type":"FILE","length":2,"owner":"ms","group":"ms","permission":"644","accessTime":1447332222908,"modificationTime":1447253745711,"blockSize":67108864,"replication":3}}
[ms at cosmosmaster-gi ~]$
Seems like Cosmos will not service connection arriving from a fiware lab
instance running on Spain2.
Does anyone knows what's the issue here, and how to correct this.
Thanks!
Menachem S.
On Nov 11, 2015 13:00, <fiware-incense-coaching-request at lists.fiware.org>
wrote:
Send Fiware-incense-coaching mailing list submissions to
fiware-incense-coaching at lists.fiware.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.fiware.org/listinfo/fiware-incense-coaching
or, via email, send a message with subject or body 'help' to
fiware-incense-coaching-request at lists.fiware.org
You can reach the person managing the list at
fiware-incense-coaching-owner at lists.fiware.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Fiware-incense-coaching digest..."
Today's Topics:
1. Heaving problems perssiting data from Cygnus to Cosmos global
instanse (Menachem Shtalhaim)
----------------------------------------------------------------------
Message: 1
Date: Tue, 10 Nov 2015 17:56:06 +0200
From: Menachem Shtalhaim <menachem.shtalhaim at n-join.com>
To: fiware-incense-coaching at lists.fiware.org
Subject: [Fiware-incense-coaching] Heaving problems perssiting data
from Cygnus to Cosmos global instanse
Message-ID:
<CAJgi6OaNek66JsYXizp7O2g=-9f8APSKbFO0d+wdQWDRgJeMQw at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
I have looked at the log and found the following:
10 Nov 2015 16:39:57,903 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistAggregation:918) -
[hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
(def_serv/def_servpath/8_registervalues/8_registervalues.txt), Data
({"recvTime":"2015-11-10T15:39:39.919Z","fiwareservicepath":"def_servpath","entityId":"8","entityType":"RegisterValues",
"value":"1959", "value_md":[], "name":"register number 8", "name_md":[]})
10 Nov 2015 16:39:58,183 ERROR
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:305) - Persistence
error (The /user/ms/def_serv/def_servpath/8_registervalues directory could
not be created in HDFS. Server response: 503 Service unavailable)
10 Nov 2015 16:39:58,183 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:307) - Finishing
transaction (1447169873-477-0000000000)
Is there anything wrong with the configuration I am using?
*Agent_1.conf:*
# Copyright 2014 Telef??nica Investigaci??n y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License
along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License
please contact with iot_support at tid dot es
#=============================================
# To be put in APACHE_FLUME_HOME/conf/agent.conf
#
# General configuration template explaining how to setup a sink of each of
the available types (HDFS, CKAN, MySQL).
#=============================================
# The next tree fields set the sources, sinks and channels used by Cygnus.
You could use different names than the
# ones suggested below, but in that case make sure you keep coherence in
properties names along the configuration file.
# Regarding sinks, you can use multiple types at the same time; the only
requirement is to provide a channel for each
# one of them (this example shows how to configure 3 sink types at the same
time). Even, you can define more than one
# sink of the same type and sharing the channel in order to improve the
performance (this is like having
# multi-threading).
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel
# source class, must not be changed
cygnusagent.sources.http-source.type =
org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming
notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler =
com.telefonica.iot.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = def_serv
# Default service path (service path semantic depends on the persistence
sink)
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
# Number of channel re-injection retries before a Flume event is definitely
discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type =
com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path
to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file =
/usr/cygnus/conf/grouping_rules.conf
# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type =
com.telefonica.iot.cygnus.sinks.OrionHDFSSink
#? true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.hdfs-sink.enable_grouping = false
# rest if the interaction with HDFS will be WebHDFS/HttpFS-based, binary if
based on the Hadoop API
cygnusagent.sinks.hdfs-sink.backend_impl = rest
# Comma-separated list of FQDN/IP address regarding the HDFS Namenode
endpoints
# If you are using Kerberos authentication, then the usage of FQDNs instead
of IP addresses is mandatory
cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fiware.org
# port of the HDFS service listening for persistence operations; 14000 for
httpfs, 50070 for webhdfs
cygnusagent.sinks.hdfs-sink.hdfs_port = 14000
# username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.hdfs_username = COSMOS_USER
#? password for the above username; this is only required for Hive
authentication
cygnusagent.sinks.hdfs-sink.hdfs_password = COSMOS_PASSWORD
# OAuth2 token for HDFS authentication
#cygnusagent.sinks.hdfs-sink.oauth2_token = TOKEN
# how the attributes are stored, available formats are json-row,
json-column, csv-row and csv-column
cygnusagent.sinks.hdfs-sink.file_format = json-column
#? number of notifications to be included within a processing batch
cygnusagent.sinks.hdfs-sink.batch_size = 100
# timeout for batch accumulation
cygunsagent.sinks.hdfs-sink.batch_timeout = 30
# Hive enabling
cygnusagent.sinks.hdfs-sink.hive = true
# Hive server version, 1 or 2 (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.server_version = 2
# Hive FQDN/IP address of the Hive server (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org
# Hive port for Hive external table provisioning (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.port = 10000
# Kerberos-based authentication enabling
cygnusagent.sinks.hdfs-sink.krb5_auth = false
# Kerberos username (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
# Kerberos password (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
# Kerberos login file (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file =
/usr/cygnus/conf/krb5_login.conf
# Kerberos configuration file (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file =
/usr/cygnus/conf/krb5.conf
#=============================================
# hdfs-channel configuration
# channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory
# capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
*CYGNUS_INSTANCE_1.CONF*
#####
#
# Configuration file for apache-flume
#
#####
# Copyright 2014 Telefonica Investigaci??n y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License
along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License
please contact with iot_support at tid dot es
# Who to run cygnus as. Note that you may need to use root if you want
# to run cygnus in a privileged port (<1024)
CYGNUS_USER=cygnus
# Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf
# Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
# Name of the agent. The name of the agent is not trivial, since it is the
base for the Flume parameters
# naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnusagent
# Name of the logfile located at /var/log/cygnus. It is important to put
the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log
# Administration port. Must be unique per instance
ADMIN_PORT=8081
# Polling interval (seconds) for the configuration reloading
POLLING_INTERVAL=30
Thanks you!
--
*Menachem Shtalhaim*
menachem.shtalhaim at n-join.com <elkana at n-join.com> | +972-5
<%2B972-527-213941>47241959 | www.n-join.com
*n-Join ? Maximizing industrial potential*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.fiware.org/pipermail/fiware-incense-coaching/attachments/20151110/4b6d91e3/attachment-0001.html
>
------------------------------
Since January 1st, old domains won't be supported and messages sent to any
domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (
Fiware-incense-coaching at lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-incense-coaching mailing list
Fiware-incense-coaching at lists.fiware.org
https://lists.fiware.org/listinfo/fiware-incense-coaching
End of Fiware-incense-coaching Digest, Vol 13, Issue 3
******************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-incense-coaching/attachments/20151113/6da685b1/attachment.html>
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy