[Fiware-lab-federation-nodes] Bug found in Kilo metadata service

[FERNANDO LOPEZ AGUILAR]

Dear all,

From time to time, currently around 2 days in Spain node, for some reasons the metadata-agent
goes into a endless or infinite loop. We could see something similar in the following identified
bugs by Mirantis https://bugs.launchpad.net/mos/+bug/1456570

The only traces that we obtained were:

2016-02-15 18:22:44.186 42222 WARNING neutron.agent.metadata.agent [-]
Server does not support metadata RPC, fallback to using neutron client
.......
2016-02-16 08:29:35.380 42224 INFO eventlet.wsgi.server [-]
192.168.220.222,<local> - - [16/Feb/2016 08:29:35] "GET
/latest/meta-data/ HTTP/1.1" 500 229 0.011348
2016-02-16 08:29:35.398 42224 INFO eventlet.wsgi.server [-] (42224)
accepted ''
2016-02-16 08:29:35.408 42224 ERROR neutron.agent.metadata.agent [-]
Unexpected error.
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
Traceback (most recent call last):
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
109, in __call__
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
204, in _get_instance_and_tenant_id
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
ports = self._get_ports(remote_address, network_id, router_id)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
192, in _get_ports
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
networks = self._get_router_networks(router_id)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 101, in
__call__
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
return self._get_from_cache(target_self, *args, **kwargs)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 79, in
_get_from_cache
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
item = self.func(target_self, *args, **kwargs)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
153, in _get_router_networks
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
internal_ports = self._get_ports_from_server(router_id=router_id)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
135, in _get_ports_from_server
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
return self._get_ports_using_client(filters)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
177, in _get_ports_using_client
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
ports = client.list_ports(**filters)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
99, in with_params
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent     ret
= self.function(instance, *args, **kwargs)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
497, in list_ports
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
**_params)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
304, in list
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent     for
r in self._pagination(collection, path, **params):
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
317, in _pagination
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent     res
= self.get(path, params=params)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
290, in get
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
headers=headers, params=params)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
267, in retry_request
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
headers=headers, params=params)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
197, in do_request
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
content_type=self.content_type())
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 158, in
do_request
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
self.authenticate_and_fetch_endpoint_url()
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 123, in
authenticate_and_fetch_endpoint_url
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
self.authenticate()
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 237, in
authenticate
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
self._authenticate_keystone()
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 218, in
_authenticate_keystone
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
raise exceptions.Unauthorized(message=resp_body)
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
Unauthorized: {"error": {"message": "The resource could not be found.",
"code": 404, "title": "Not Found"}}
2016-02-16 08:29:35.408 42224 TRACE neutron.agent.metadata.agent
2016-02-16 08:29:35.412 42224 INFO eventlet.wsgi.server [-]
192.168.220.222,<local> - - [16/Feb/2016 08:29:35] "GET
/latest/meta-data/ HTTP/1.1" 500 229 0.012526
2016-02-16 08:29:35.430 42224 INFO eventlet.wsgi.server [-] (42224)
accepted ''
.............

We have identified a workaround to recover the service, just restart the
metadata-agent and the service continue working, but we do not know
The reason why it is produced this error.

Fernando.-


________________________________

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-lab-federation-nodes/attachments/20160216/ab9620b1/attachment.html>