Hi Craig,
Many thanks for this valuable input.
Some responses inline …
De: Craig Aldridge <caldridge at gsma.com<mailto:caldridge at gsma.com>>
Fecha: lunes, 10 de agosto de 2015, 15:23
Para: Jose Manuel Cantera Fonseca <josemanuel.canterafonseca at telefonica.com<mailto:josemanuel.canterafonseca at telefonica.com>>, "fiware-ngsi at lists.fi-ware.eu<mailto:fiware-ngsi at lists.fi-ware.eu>" <fiware-ngsi at lists.fi-ware.eu<mailto:fiware-ngsi at lists.fi-ware.eu>>
CC: MARCOS REYES UREÑA <marcos.reyesurena at telefonica.com<mailto:marcos.reyesurena at telefonica.com>>, "denny.kim at kt.com<mailto:denny.kim at kt.com>" <denny.kim at kt.com<mailto:denny.kim at kt.com>>, Ben Howes <BHowes at gsma.com<mailto:BHowes at gsma.com>>, Stephen Doyle <sdoyle at gsma.com<mailto:sdoyle at gsma.com>>
Asunto: RE: [Fiware-ngsi] GSMA contribution: New "Historic" API suggestions
Dear All,
Jose thanks very much for taking the time to prepare these example queries for the group. There are a several points that the GSMA needs to raise:
1. Steve has raised an issue with the suggested naming as the ‘historical’ API as this is meant to be a general purpose API for high volume/ frequently changing data. Therefore we would like to refer to this branch of the API rather as the ‘dynamic’ API to ensure it is not perceived as a legacy/ redundant API.
>> Well, the name is not that important … we are open to better names if we find something accurate and not misleading
2. The examples you provided seem to be particularly optimal for SQL, which of course we are aware are Telefonica’s preferred implementation strategy for the “dynamic” (Historic) API. However SQL is not GSMA’s recommendation for the high / changeable volume data that will feature in live IoT applications and traditional RDBMS systems are simply not an effective technology to use in high volume data warehouse applications that we expect to be the case with IoT. Given the requirement for high efficiency ad-hoc joins we don’t think that examples which are optimised for traditional RDBMS systems are relevant to NoSQL storage or graph/ document backends. In addition we are concerned that the suggested query language is following RDBMS specific paradigms and were working on the basis of ensuring we were all trying to create a platform which is technology agnostic and of course efficient for IoT applications.
>> We strongly disagree here. We are defining a Query Language that can be implemented on top of NoSQL or other technologies. You still have to demonstrate that is not implementable on top of NoSQL
3. The lack of unique entity references at the database level means that the complexity of implementing graph queries is pushed on to the developer, who needs to understand the relationships between data he wishes to mash up, rather than being able to follow simple semantic references. This will also lead to significant system/ application inefficiencies due to the requirement for applications to make large numbers of requests when accessing or querying across entities. Whilst obviously simplifying the IoT platform this seems a significant step backwards in what the project is trying to do in terms of creating an attractive ecosystem.
>> Could you suggest an internal data model, implementable, materialized, that could solve the problem on a NoSQL database and which could hide the complexity of such a data model to the developer? We like your proposal but we would like it to be concretized and materialized in order to commit to it. We are open to some syntactic sugar that could hide some of the underlying complexities of the internal data model, but not at the prize of being too inflexible … I think something in the middle is possible though …
>From the examples, there are a couple of questions too:
4. It's not clear from your examples how you handle non-trivial geographic queries? e.g. in your example, are you attempting to perform an exact match on a simple point+radius search area, or an inclusion? It has been agreed previously that a complete set of Geographic query features was a requirement.
>> Inclusion is implicit and more language machinery could be added if needed for more advanced Geo operators.
5. It's not clear if your spec meets the agreed Rich operators requirement which was agreed? It would be useful to see examples of matching against a list of choices and regular expression type operators.
>> You can add more operators to the condition part of the query object, particularly those already defined in the Simple Query Language
Referring back to previous communications from GSMA – particularly the email from Steve – we are willing to request implementation of an improved version of NGSI2 for the GSMA sandbox provided that the specification addresses the specific needs that GSMA has identified to support commercially live IoT applications. We know that the current NGSI2 specification could be used now to create purely demonstration applications but we cannot in reality engage with external developers with a specification we feel to be difficult to use or inefficient for a variety of commercially live applications.
>> That still has to be demonstrated … however what we are proposing is pragmatic, materialized and implementable … your MQLv2 proposal was not in our opinion … Furthermore, and that’s my particular opinion, your score system for Query Languages was a bit biased towards MQL instead of being more neutral.
Earlier in the week we were feeling that some of the compromises that had been made by GSMA were being met with acceptance but we are disappointed that subsequently GSMA’s constructive technical input has been dismissed.
>> That is not true. Your input has been adapted and materialized to something implementable … We have accepted complex queries as a requirement and we have defined a Query JSON object that meets the requirements we agreed beforehand. The ideas behind the MQL language are still there and I think our proposal leverages MQLv2 as overcomes its multiple flaws.
Again we should stress that we will only implement a technology in the GSMA sandbox that we feel is fit for purpose in IoT applications.
>> Here is the only point on which I fully agree with you. We are not going to waste our precious implementation resources on devising something is not going to work.
We welcome further feedback on our points raised above from both Telefonica and other Fiware participants.
>> Thanks for your contributions, they are really helpful and I think we will finally come up with something awesome!
Many thanks,
Craig Aldridge | Project Manager | IoT Big Data Ecosystem|GSMA | +44 7725 221 299
GSMA, Floor 2, The Walbrook Building, 25 Walbrook, London, EC4N 8AF
PPlease consider the environment – don’t print this mail unless you really need to...
From: fiware-ngsi-bounces at lists.fi-ware.org<mailto:fiware-ngsi-bounces at lists.fi-ware.org> [mailto:fiware-ngsi-bounces at lists.fi-ware.org] On Behalf Of JOSE MANUEL CANTERA FONSECA
Sent: 07 August 2015 12:31
To: fiware-ngsi at lists.fi-ware.eu<mailto:fiware-ngsi at lists.fi-ware.eu>
Cc: MARCOS REYES UREÑA <marcos.reyesurena at telefonica.com<mailto:marcos.reyesurena at telefonica.com>>; denny.kim at kt.com<mailto:denny.kim at kt.com>; Ben Howes <BHowes at gsma.com<mailto:BHowes at gsma.com>>
Subject: Re: [Fiware-ngsi] GSMA contribution: New "Historic" API suggestions
Dear all,
As a follow-up action of the email below, please find below a link to Telefonica's proposal for a JSON payload to express queries intended to the Historical Context (HAPI). We have been inspired by the nice commencement provided by GSMA, MQLv2, and leveraged it as we stated in previous communications. The result is here:
https://docs.google.com/document/d/1L_7uqFfYxTkvzx--lzs1NbdWOBxpRy98hsnHBx07WB4/edit?usp=sharing
This proposal meets the requirements we described below, namely:
A/ Separation of concerns. We have separated
* the query itself (q field)
* the content of the array of results (result field)
* The options for the query (pagination, etc.)
Regarding the query, we have separated aspects as well:
* The affected entity types (or ids) of the model, combination of entities is allowed, enabling cartesian products between entities
* The filter, which includes three dimensions: data dimension, spatial dimension and temporal dimension
* The orderBy statements. (if any)
Regarding the result a simple and unambiguous notation and functions to map JSON-result fields with entity data fields has been defined.
Concerning the condition (data dimension), SQL-Like expressions are allowed and developers can express them naturally. Relationships between model entities are not hardcoded but made explicit simplifying implementation requirements and at the same time enabling high flexibility and query serendipity (note that we will not only be working with standard data models but with different data models where the relationships between entities can be made in unforeseen, not a priori defined manners)
We believe the proposed JSON payload is a significant step forward, and more importantly it has been designed taking into account its implementability. However, more work has to be done and your contributions, to fine tune and refine it, are welcome.
Questions are welcome as well
Many thanks, have a good weekend.
All the best
Jose Manuel
De: <fiware-ngsi-bounces at lists.fi-ware.org<mailto:fiware-ngsi-bounces at lists.fi-ware.org>> on behalf of Jose Manuel Cantera Fonseca <josemanuel.canterafonseca at telefonica.com<mailto:josemanuel.canterafonseca at telefonica.com>>
Fecha: jueves, 6 de agosto de 2015, 10:03
Para: "fiware-ngsi at lists.fi-ware.eu<mailto:fiware-ngsi at lists.fi-ware.eu>" <fiware-ngsi at lists.fi-ware.eu<mailto:fiware-ngsi at lists.fi-ware.eu>>
CC: MARCOS REYES UREÑA <marcos.reyesurena at telefonica.com<mailto:marcos.reyesurena at telefonica.com>>, Ben Howes <BHowes at gsma.com<mailto:BHowes at gsma.com>>
Asunto: Re: [Fiware-ngsi] GSMA contribution: New "Historic" API suggestions
Thanks Craig and Ben for this contribution, is a considerable step forward, however we have two major concerns on the proposed MQLv2 language that I think would deserve a careful analysis by the group. That would mean that your proposed deadlines are going to be extended. We don’t think that would be a problem, a problem would be to define a specification which is inconsistent or flawed since it was born and what it is worst, unimplementable using conventional technologies for instance, RDBMS.
These concerns have been derived from careful analysis and by considering an experiment like the one which is described below:
Experiment
A simple data model of TrafficIncidences and Roads was defined. A 'traffic incidence' happens in a Road and a Road can be affected by many traffic incidences.
Then a query like “Tell me the cause, road capacity" of the Traffic Incidences which happened on roads which severity was high or cause was ‘Vehicle collision’ and road capacity was greater than 2”
We asked different engineers to express the query using SQL-like conventions and using the aforementioned MQLv2. All of them were able to describe the query very quickly in SQL terms
Select cause, capacity
>From TrafficIncidence, Road
Where Road.id = TrafficIncidence.road
AND severity = ‘high’ OR (cause = ‘Vehicle Collision’ and capacity > 2)
but were in a trouble coming up to an expression like
{
“type”: “TrafficIncidence”,
“severity”: “high",
“cause”: null,
“road”: {
“capacity”: null
} ,
“$or”: {
“cause”: “Vehicle Collision”,
“road”: {
“capacity > “: 2
}
}
}
Our experiment confirmed that the usability and developer friendliness of representing complex filters (AND, OR, etc.) using a JSON tree is essentially very bad. Furthermore it hardly meets the DRY principle (Don’t Repeat Yourself). On the other hand it does not meet the principle of separation of concerns as it mixes in the same structure and conventions the three different aspects a query must have, which are:
1/ What entity (or entity types) the query operates on
2/ What are the selected attributes
3/ What are the filter conditions
So even though at first glance the MQLv2 syntax might seem nice at the end of the day it might turn into a double-edge sword. For instance, am I looking for traffic incidences which cause is unknown (null) or do I want the cause attribute in the query result objects? Or is severity going to be included in the result or not?
So the MQLv2 language is ambiguous and difficult to read and write.
Second major concern ‘Implementability” . So how an implementation, which has to be “generic” and agnostic on the underlying data models is going to be able to resolve queries if the only information which is passed is a JSON structure like the one described above. How an implementation will know that the road entity and the TrafficIncidence entity are going to be linked by the road.id and TrafficIncidence.road fields? Of course, a dummy implementation which has the data models “hardcoded” and some “a priori" knowledge can perform the join automatically but a generic implementation which is data model agnostic is not going to be able to do so.
So the MQLv2 language is difficult to implement by generic databases on conventional technologies, namely RDBMS.
These are the two major concerns we have with MQLv2 language and which have to be addressed, before any spec work can start.
Trying to summarize, separation of concerns, implementability and readability are three key areas on which we want to work in order to craft a specification consistent, simple and powerful.
During the next days, hopefully by the end of the week, we will be sending a proposal (JSON-based) that leverages the MQLv2, and which we think will meet those paramount requirements.
I hope this clarifies, thank you one more time for your contributions, and looking forward to working with you in finding a solution which can meet everybody’s needs.
All the best
De: <fiware-ngsi-bounces at lists.fi-ware.org<mailto:fiware-ngsi-bounces at lists.fi-ware.org>> on behalf of Craig Aldridge <caldridge at gsma.com<mailto:caldridge at gsma.com>>
Fecha: miércoles, 5 de agosto de 2015, 17:21
Para: "fiware-ngsi at lists.fi-ware.eu<mailto:fiware-ngsi at lists.fi-ware.eu>" <fiware-ngsi at lists.fi-ware.eu<mailto:fiware-ngsi at lists.fi-ware.eu>>
CC: Ben Howes <BHowes at gsma.com<mailto:BHowes at gsma.com>>
Asunto: [Fiware-ngsi] GSMA contribution: New "Historic" API suggestions
Dear all,
Following on from the GSMA’s previous communication on suggested contributions to the NGSIv2 specification, we have been discussing in detail and have agreed with Telefonica a proposedway forward. It is felt that the best way forward is todevelop a new API that will deal with complex and historical queries rather than make breaking changes to the existing draft of the NGSIv2 specification. As such, please find detailed below (and attached) our thoughts on the new ‘Historic’ API.
GSMA have evaluated several choices of language for the historic API query language by means of a weighted ranking matrix. The criteria which each query DSL was ranked by was based on requirements defined in collaboration with the FIWARE team at Telefonica. MQL and elastic search are the two main contenders as a result of this, however neither is perfectly suited.
GSMA had already been working to create a specification (which you will see as MQLv2 in the ranking matrix) prior to involvement with FIWARE, which we feel offers a great choice of DSL to meet all the current requirements for querying, as well as offering scope to expand features in the future without breaking existing queries.
The attached slides provide an overview of the 3 key areas GSMA have sought to improve in MQL, whilst the document shows in more detail how these improvements look from a more technical perspective, as well as some examples of these features in use.
Can I please ask that this group review our suggestions and provide any feedback by no later than close of business on the 11th August; our target is to provide a full specification of the MQLv2 specification to this group by 14th August for final review.
If you have any questions please feel free to be contact me,
Kind regards,
Craig Aldridge | Project Manager | IoT Big Data Ecosystem|GSMA | +44 7725 221 299
GSMA, Floor 2, The Walbrook Building, 25 Walbrook, London, EC4N 8AF
PPlease consider the environment – don’t print this mail unless you really need to...
This email and its attachments are intended for the above named only and may be confidential. If they have come to you in error you must take no action based on them, nor must you copy or show them to anyone; please reply to this email or call +44 207 356 0600 and highlight the error.
________________________________
Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
________________________________
Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
This email and its attachments are intended for the above named only and may be confidential. If they have come to you in error you must take no action based on them, nor must you copy or show them to anyone; please reply to this email or call +44 207 356 0600 and highlight the error.
________________________________
Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-ngsi/attachments/20150810/506ccba1/attachment.html>
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy