[ https://jira.fiware.org/browse/HELP-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
José Ignacio Carretero Guarde resolved HELP-9184.
-------------------------------------------------
Resolution: Done
> [fiware-stackoverflow] Cygnus not presisting data on MySql database
> -------------------------------------------------------------------
>
> Key: HELP-9184
> URL: https://jira.fiware.org/browse/HELP-9184
> Project: Help-Desk
> Issue Type: Monitor
> Components: FIWARE-TECH-HELP
> Reporter: Backlog Manager
> Assignee: Francisco Romero
> Labels: fiware, fiware-cygnus, fiware-orion, mysql
>
> Created question in FIWARE Q/A platform on 28-05-2015 at 17:05
> {color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/30511082/cygnus-not-presisting-data-on-mysql-database
> +Question:+
> Cygnus not presisting data on MySql database
> +Description:+
> So i have read all the documentation and followed the tutorial on MySQL persistence but i can't still presist any kind of data on MySQL database.
> Even though i'm puting the presistence mode = row it doesn't create any database nor table.
> What am i doing wrong?
> My Subscription:
> python2.7 SetSubscription.py bustest4 http://localhost:5050/notify
> Output:
> * Asking to http://localhost:1026/v1/subscribeContext
> * Headers: {'Fiware-Service': 'fiwaretestapi', 'content-type': 'application/json ', 'accept': 'application/json', 'X-Auth-Token': 'NULL'}
> * Sending PAYLOAD:
> {
> "reference": "http://localhost:5050/notify",
> "throttling": "PT5S",
> "entities": [
> {
> "type": "",
> "id": "bustest4",
> "isPattern": "false"
> }
> ],
> "attributes": [
> "temperature"
> ],
> "duration": "P1M",
> "notifyConditions": [
> {
> "condValues": [
> "temperature"
> ],
> "type": "ONCHANGE"
> }
> ]
> }
> ...
> * Status Code: 200
> {
> "subscribeResponse" : {
> "subscriptionId" : "5567332298add18cc3e183ac",
> "duration" : "P1M",
> "throttling" : "PT5S"
> }
> }
> My agent_a1.conf:
> #=============================================
> # source configuration
> # channel name where to write the notification events
> cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
> # source class, must not be changed
> cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
> # listening port the Flume source will use for receiving incoming notifications
> cygnusagent.sources.http-source.port = 5050
> # Flume handler that will parse the notifications, must not be changed
> cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
> # URL target
> cygnusagent.sources.http-source.handler.notification_target = /notify
> # Default service (service semantic depends on the persistence sink)
> cygnusagent.sources.http-source.handler.default_service = fiwaretestapi
> # Default service path (service path semantic depends on the persistence sink)
> cygnusagent.sources.http-source.handler.default_service_path = /
> # Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
> cygnusagent.sources.http-source.handler.events_ttl = 10
> # Source interceptors, do not change
> cygnusagent.sources.http-source.interceptors = ts de
> # Timestamp interceptor, do not change
> cygnusagent.sources.http-source.interceptors.ts.type = timestamp
> # Destination extractor interceptor, do not change
> cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
> # Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
> # See the doc/design/interceptors document for more details
> cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf
> # ============================================
> # OrionMySQLSink configuration
> # channel name from where to read notification events
> cygnusagent.sinks.mysql-sink.channel = mysql-channel
> # sink class, must not be changed
> cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
> # the FQDN/IP address where the MySQL server runs
> cygnusagent.sinks.mysql-sink.mysql_host = localhost
> # the port where the MySQL server listes for incomming connections
> cygnusagent.sinks.mysql-sink.mysql_port = 3306
> # a valid user in the MySQL server
> cygnusagent.sinks.mysql-sink.mysql_username = root
> # password for the user above
> cygnusagent.sinks.mysql-sink.mysql_password = ***********
> # how the attributes are stored, either per row either per column (row, column)
> cygnusagent.sinks.mysql-sink.attr_persistence = row
> #=============================================
> # mysql-channel configuration
> # channel type (must not be changed)
> cygnusagent.channels.mysql-channel.type = memory
> # capacity of the channel
> cygnusagent.channels.mysql-channel.capacity = 1000
> # amount of bytes that can be sent per transaction
> cygnusagent.channels.mysql-channel.transactionCapacity = 100
> My cygnus_instance_c1.conf:
> # Who to run cygnus as. Note that you may need to use root if you want
> # to run cygnus in a privileged port (<1024)
> CYGNUS_USER=root
> # Where is the config folder
> CONFIG_FOLDER=/usr/cygnus/conf
> # Which is the config file
> CONFIG_FILE=/usr/cygnus/conf/agent_a1.conf
> # Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters
> # naming conventions, e.g. it appears in .sources.http-source.channels=...
> AGENT_NAME=cygnususer
> # Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
> LOGFILE_NAME=cygnus.log
> # Administration port. Must be unique per instance
> ADMIN_PORT=8081
> My cygnus.log:
> Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
> Warning: JAVA_HOME is not set!
> + exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication -p 8081 -f /usr/cygnus/conf/agent_a1.conf -n root
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> EDIT:
> So after some changes i got the log file to work and i found out that the 8081 port was already in use. Can you explain me what the ADMIN_PORT is used for, what port is it recommended to be?
> LOG FILE:
> 02 Jun 2015 05:16:40,680 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/cygnus/conf/agent_a1.conf
> 02 Jun 2015 05:16:40,685 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
> 02 Jun 2015 05:16:40,686 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
> 02 Jun 2015 05:16:40,686 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930) - Added sinks: mysql-sink Agent: cygnusagent
> 02 Jun 2015 05:16:40,686 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
> 02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
> 02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
> 02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
> 02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
> 02 Jun 2015 05:16:40,692 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [cygnusagent]
> 02 Jun 2015 05:16:40,692 WARN [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:138) - No configuration found for this host:root
> 02 Jun 2015 05:16:40,693 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
> 02 Jun 2015 05:16:40,693 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
> 02 Jun 2015 05:16:40,694 INFO [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85) - Starting a Jetty server listening on port 8081 (Management Interface)
> 02 Jun 2015 05:16:40,695 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
> 02 Jun 2015 05:16:40,695 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
> 02 Jun 2015 05:16:40,695 INFO [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85) - Starting a Jetty server listening on port 8081 (Management Interface)
> 02 Jun 2015 05:16:40,696 INFO [Thread-26] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
> 02 Jun 2015 05:16:40,704 WARN [Thread-26] (org.mortbay.log.Slf4jLog.warn:76) - failed SocketConnector at 0.0.0.0:8081: java.net.BindException: Address already in use
> 02 Jun 2015 05:16:40,704 WARN [Thread-26] (org.mortbay.log.Slf4jLog.warn:76) - failed Server at 4f1b95f: java.net.BindException: Address already in use
> 02 Jun 2015 05:16:40,704 FATAL [Thread-26] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63) - Fatal error running the Management Interface. Details=Address already in use
> 02 Jun 2015 05:16:40,705 INFO [Thread-27] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
> 02 Jun 2015 05:16:40,709 WARN [Thread-27] (org.mortbay.log.Slf4jLog.warn:76) - failed SocketConnector at 0.0.0.0:8081: java.net.BindException: Address already in use
> 02 Jun 2015 05:16:40,709 WARN [Thread-27] (org.mortbay.log.Slf4jLog.warn:76) - failed Server at ed4c222: java.net.BindException: Address already in use
> 02 Jun 2015 05:16:40,709 FATAL [Thread-27] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63) - Fatal error running the Management Interface. Details=Address already in use
> EDIT 2:
> My script that updates entity on Context Broker:
> BASE_URL = 'http://localhost:1026'
> UPDATE_URL = BASE_URL+'/ngsi10/updateContext'
> HEADERS = {
> 'Content-Type': 'application/json',
> 'Accept': 'application/json',
> 'Fiware-Service' : 'fiwaretestapi',
> 'Fiware-ServicePath': '/'
> }
> UPDATE_EXAMPLE = {
> "contextElements": [
> {
> "type": "",
> "isPattern": "false",
> "id": "bustest4",
> "attributes": [
> {
> "name": "temperature",
> "type": "int",
> "value": "99"
> }
> ]
> }
> ],
> "updateAction": "APPEND"
> }
> def post(url, data):
> """"""
> req = urllib2.Request(url, data, HEADERS)
> f = urllib2.urlopen(req)
> result = json.loads(f.read())
> f.close()
> return result
> if __name__ == "__main__":
> print post(UPDATE_URL, json.dumps(UPDATE_EXAMPLE))
> EDIT3:
> Even though i set the admin port to be 8085 on the cygnus agent configuration it tries to bind to the 8081, is that normal?
> Here are the logs from the cygnus:
> time=2015-06-12T05:56:39.820EDT | lvl=INFO | trans= | function=start | comp=Cygnu s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty pe: CHANNEL, name: mysql-channel started
> time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents | comp=Cygnus | msg=org.apache.flume.node.Application[173] : Starting Sink mysql -sink
> time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents | comp=Cygnus | msg=org.apache.flume.node.Application[184] : Starting Source htt p-source
> time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startManagementInt erface | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusAppl ication[85] : Starting a Jetty server listening on port 8081 (Management Interfac e)
> time=2015-06-12T05:56:39.822EDT | lvl=INFO | trans= | function=start | comp=Cygnu s | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink[151] : [mysql- sink] Startup completed
> time=2015-06-12T05:56:39.823EDT | lvl=INFO | trans= | function=info | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
> time=2015-06-12T05:56:39.824EDT | lvl=INFO | trans= | function=info | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
> time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=info | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[67] : Started SocketConnector at 0.0.0.0:5050
> time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=start | comp=Cygnu s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty pe: SOURCE, name: http-source started
> time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[76] : failed SocketConnector at 0.0.0.0:8081: java.n et.BindException: Address already in use
> time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[76] : failed Server at 1c9c5521: java.net.BindExcept ion: Address already in use
> time=2015-06-12T05:56:39.827EDT | lvl=FATAL | trans= | function=run | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer[63] : Fatal error r unning the Management Interface. Details=Address already in use
> Log when i make a subscription:
> time=2015-06-12T06:03:56.529EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle rs.OrionRestHandler[153] : Starting transaction (1434103313-190-0000000000)
> time=2015-06-12T06:03:56.535EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle rs.OrionRestHandler[239] : Received data ({ "subscriptionId" : "557aae8c98add18c c3e183b6", "originator" : "localhost", "contextResponses" : [ { "contex tElement" : { "type" : "thing", "isPattern" : "false", "id" : "autocarro1", "attributes" : [ { "name" : "temperatu re", "type" : "int", "value" : "95", "metadatas" : [ { "name" : "TimeInstant", "type" : "ISO8601", "value" : "2015-06-03T09:17:44.046583Z" } ] } ] }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
> time=2015-06-12T06:03:56.540EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle rs.OrionRestHandler[261] : Event put in the channel (id=1983722072, ttl=10)
> time=2015-06-12T06:03:56.724EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or ionSink[126] : Event got from the channel (id=1983722072, headers={timestamp=1434 103436542, content-type=application/json, transactionId=1434103313-190-0000000000 , fiware-service=fiwaretestapi, fiware-servicepath=, ttl=10, destination=autocarr o1_thing}, bodyLength=657)
> time=2015-06-12T06:03:57.260EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=persist | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or ionMySQLSink[227] : [mysql-sink] Persisting data at OrionMySQLSink. Database: fiw aretestapi, Table: autocarro1_thing, Data: 1434103436,2015-06-12T06:03:56.542,aut ocarro1,thing,temperature,thing,95,[{"name":"TimeInstant","type":"ISO8601","value ":"2015-06-03T09:17:44.046583Z"}]
> time=2015-06-12T06:03:57.270EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or ionSink[187] : Finishing transaction (1434103313-190-0000000000)
--
This message was sent by Atlassian JIRA
(v6.4.1#64016)
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy