[Fiware-ngsi] Handling query/discover responses with very large results

[Salvatore Longo]

Hi Fermin,
I agree with you that a huge response cannot handle via network layer and also the performance of the component are related to how big is the response body.  I like the twitter approach and I think could be interesting include inside the URL some optional parameters like twitter does for specifying how many queryContext are expected .


-          Salvatore Longo


________________________________

Salvatore Longo
Software Engineer
NEC Europe Ltd.
Kurfürsten-Anlage 36
D-69115 Heidelberg

Tel.  +49/(0) 62 21/43 42 - 246
Fax. +49/(0) 62 21/43 42 - 115
E-Mail: Salvatore.longo at neclab.eu

NEC Europe Limited
Registered Office: NEC House,
1 Victoria Road, London W3 6BL
Registered in England 2832014

________________________________

From: fiware-ngsi-bounces at lists.fi-ware.eu [mailto:fiware-ngsi-bounces at lists.fi-ware.eu] On Behalf Of Fermín Galán Márquez
Sent: Mittwoch, 27. Februar 2013 17:50
To: fiware-ngsi at lists.fi-ware.eu
Subject: [Fiware-ngsi] Handling query/discover responses with very large results

Hi,

In large-scale utilization scenarios (e.g. 100,000 entities registered) a NGSI10 queryContext with too wide patterns (e.g. the ".*" that matches everything) could be overwhelming (e.g. it could involve an XML payload containing 100,000 entities information in the HTTP response to the client). A similar case could happen with NGSI9 discoverContextAvailability operation.

We should provide a solution for these cases:

  *   In the short term (i.e. for Release 2), we could just define a new error code (e.g. "483 Response Too Long") that the NGSI server could use if a given limit is exceeded (and particular limit will be implementation specific).
  *   In the long term, probably we should define a smarter mechanism. Looking to other populars REST APIs one trend is to use paging, e.g. the Twitter API (https://dev.twitter.com/docs/api/1/get/search) uses "rpp" and "page" to specify how many items to include in the response, maybe a similar mechanism could be introduced in NGSI.

What do you think?

Best regards,

------
Fermín

________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-ngsi/attachments/20130301/9a2c08dd/attachment.html>