[Backlogmanager] [FIWARE-JIRA] (HELP-9314) [fiware-stackoverflow] HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported and Cosmos HttpFS response: HTTP/1.1 401 Unauthorized

[José Ignacio Carretero Guarde (JIRA)]

     [ https://jira.fiware.org/browse/HELP-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

José Ignacio Carretero Guarde updated HELP-9314:
------------------------------------------------
    Status: In Progress  (was: Open)

> [fiware-stackoverflow] HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported and Cosmos HttpFS response: HTTP/1.1 401 Unauthorized
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HELP-9314
>                 URL: https://jira.fiware.org/browse/HELP-9314
>             Project: Help-Desk
>          Issue Type: Monitor
>          Components: FIWARE-TECH-HELP
>            Reporter: Backlog Manager
>            Assignee: Francisco Romero
>              Labels: fiware, fiware-cosmos, fiware-cygnus, fiware-orion
>
> Created question in FIWARE Q/A platform on 17-07-2015 at 15:07
> {color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/31477545/httpbadrequestexception-orion-0-19-0-libcurl-7-19-7-user-agent-not-supported-an
> +Question:+
> HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported and Cosmos HttpFS response: HTTP/1.1 401 Unauthorized
> +Description:+
> Good afternoon,
> I'm trying to configure persistence of Orion Context's in the cosmos HDFS. So far I have created a Orion Context Broker Instance (orion-psb-image-R4.2), created a Cosmos user account and the folder 'cardio' inside it (/user/USERNAME/cardio).
> I've also configured the cygnus.conf file (/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf) to be the following:
> # APACHE_FLUME_HOME/conf/cygnus.conf
> orionagent.sources = http-source
> orionagent.sinks = hdfs-sink
> orionagent.channels = notifications
> # Flume source, must not be changed
> orionagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
> # channel name where to write the notification events
> orionagent.sources.http-source.channels = notifications
> # listening port the Flume source will use for receiving incoming notifications
> orionagent.sources.http-source.port = 5050
> # Flume handler that will parse the notifications, must not be changed
> orionagent.sources.http-source.handler = es.tid.fiware.orionconnectors.cosmosinjector.OrionRestHandler
> # regular expression for the orion version the notifications will have in their headers
> orionagent.sources.http-source.handler.orion_version = 0\.11\.*
> # URL target
> orionagent.sources.http-source.handler.notification_target = /notify
> # channel name from where to read notification events
> orionagent.sinks.hdfs-sink.channel = notifications
> # Flume sink that will process and persist in HDFS the notification events, must not be changed
> orionagent.sinks.hdfs-sink.type = es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink
> # IP address of the Cosmos deployment where the notification events will be persisted
> orionagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
> # port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
> orionagent.sinks.hdfs-sink.cosmos_port = 14000
> # username allowed to write in HDFS (/user/myusername)
> orionagent.sinks.hdfs-sink.cosmos_username = USERNAME
> # dataset where to persist the data (/user/myusername/mydataset)
> orionagent.sinks.hdfs-sink.cosmos_dataset = cardio
> # HDFS backend type (webhdfs, httpfs or infinity)
> orionagent.sinks.hdfs-sink.hdfs_api = httpfs
> # channel name
> orionagent.channels.notifications.type = memory
> # capacity of the channel
> orionagent.channels.notifications.capacity = 1000
> # amount of bytes that can be sent per transaction
> orionagent.channels.notifications.transactionCapacity = 100
> The issue is that when I start the agent with the following command:
> /usr/local/apache-flume-1.4.0-bin/bin/flume-ng agent --conf /usr/local/apache-flume-1.4.0-bin/conf/ -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent -Dflume.root.logger=INFO,consoleapache-flume-1.4.0-bin
> I get the following error messages:
> 2015-07-17 15:33:10,477 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)] HttpFS operation: PUT http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&user.name=arlourenco HTTP/1.1
> 2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:71)] HttpFS response: HTTP/1.1 401 Unauthorized
> 2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [ERROR - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] The  directory could not be created in HDFS. HttpFS response: 401 Unauthorized
> 2015-07-17 15:33:17,470 (910628961 at qtp-586740701-0) [WARN - org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:186)] Received bad request from client.
> And:
> org.apache.flume.source.http.HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported
> Here is the complete log:
> + exec /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/usr/local/apache-flume-1.4.0-bin/conf:/usr/local/apache-flume-1.4.0-bin/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent
> 2015-07-17 15:33:09,858 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
> 2015-07-17 15:33:09,863 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf
> 2015-07-17 15:33:09,871 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: hdfs-sink Agent: orionagent
> 2015-07-17 15:33:09,871 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
> 2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
> 2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
> 2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
> 2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
> 2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
> 2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
> 2015-07-17 15:33:09,873 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
> 2015-07-17 15:33:09,885 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [orionagent]
> 2015-07-17 15:33:09,885 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
> 2015-07-17 15:33:09,899 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel notifications type memory
> 2015-07-17 15:33:09,908 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel notifications
> 2015-07-17 15:33:09,912 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
> 2015-07-17 15:33:09,961 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: hdfs-sink, type: es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink
> 2015-07-17 15:33:09,967 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel notifications connected to [http-source, hdfs-sink]
> 2015-07-17 15:33:09,972 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:http-source,state:IDLE} }} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor at e8234ec counterGroup:{ name:null counters:{} } }} channels:{notifications=org.apache.flume.channel.MemoryChannel{name: notifications}} }
> 2015-07-17 15:33:09,973 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel notifications
> 2015-07-17 15:33:10,027 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: CHANNEL, name: notifications, registered successfully.
> 2015-07-17 15:33:10,028 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: CHANNEL, name: notifications started
> 2015-07-17 15:33:10,028 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink hdfs-sink
> 2015-07-17 15:33:10,029 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source http-source
> 2015-07-17 15:33:10,069 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2015-07-17 15:33:10,149 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26
> 2015-07-17 15:33:10,306 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started SocketConnector at 0.0.0.0:5050
> 2015-07-17 15:33:10,307 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: SOURCE, name: http-source, registered successfully.
> 2015-07-17 15:33:10,307 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: SOURCE, name: http-source started
> 2015-07-17 15:33:10,456 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.http.HttpClientFactory.<init>(HttpClientFactory.java:68)] Setting max total connections to 500 and default max connections per route to 100
> 2015-07-17 15:33:10,477 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)] HttpFS operation: PUT http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&user.name=arlourenco HTTP/1.1
> 2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:71)] HttpFS response: HTTP/1.1 401 Unauthorized
> 2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [ERROR - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] The  directory could not be created in HDFS. HttpFS response: 401 Unauthorized
> 2015-07-17 15:33:17,470 (910628961 at qtp-586740701-0) [WARN - org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:186)] Received bad request from client.
> org.apache.flume.source.http.HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported
>         at es.tid.fiware.orionconnectors.cosmosinjector.OrionRestHandler.getEvents(OrionRestHandler.java:93)
>         at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:184)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:326)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>         at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> What could be the issue? 



--
This message was sent by Atlassian JIRA
(v6.4.1#64016)