Hallo Tobias,
Please find below some responses to the issues raised. Many thanks.
[TJ] Assume you have a context entity representing a street, and the co2-level of the street is measures by two different devices. In the NGSI model, the attribute “co2level” would then have two relevant values, each delivered by a distinct device, and these two values would be distinguished by distinct value IDs.
In FIWARE NGSI 1, applications who wish to retrieve only a specific attribute value could
- make a queryContextRequest and add to it an XPATH restriction which would limit the response to the attribute value instance having the desired value ID as metadata
- or use the convenience function making a GET on {ngsi10path}/contextEntities/myStreet/attributes/co2level/id1
A query not specifying the ID would, in contrast, return two attribute values.
In FIWARE NGSI 2 we can discuss whether the possibility to query a specific value needs to be included in the convenience function as a sub-resource of resources representing attributes – maybe this not necessary, and specific value instances can still be queried by using the corresponding restriction [restrictions are another question to discuss].
However, conceptually the NGSI system should distinguish between several value instances, so if two attribute values with different ID metadata exist, then e.g. an update on one instance should not overwrite the other.
I’m sympathetic to what you are proposing but I was wondering if what you are pursuing might result in API over engineering just to support a use case that can be seamlessly handled by the application itself. For instance, you can have a CO2 level value which can be obtained from the contribution of several measurements coming from different sensors . You might even automatically update the resulting value when a new measurement arrives by using the CEP. For instance.
{
id: “Friedrichstrasse”,
type: “Street”,
co2Level: 0.1
}
{
id: “Sensor-789”,
type: “CO2Sensor”,
location: “ Friedrichstrasse”,
measurement: 0.15
}
{
id: “Sensor-900”,
type: “CO2Sensor”,
location: “ Friedrichstrasse”,
measurement: 0..09
}
What I meant was that now an entity looking like
{
id: myStreet
co2Level: “aLot”
}
is not allowed, but needs to be something like
{
id: myStreet
co2Level: {
value: ”aLot”
type: “fun-string”
}
}
So the actual value is of JSON type string, but needs to be nested inside a JSON object where also the value type is declared.
At least this is how I interpret the “JSON entity representation” description in the ngsi2 draft, but maybe I still get it wrong?
{
id: myStreet
co2Level: “aLot”
}
In this variant there is no NGSI type informed. In the variant which include type you are informing a NGSI type but from the point of view of the implementation they are both strings.
[TJ] I need to further discuss with my colleagues about this before agreeing or proposing an alternative.
One concern is that not having the status code in the response message forces applications to also look into the response header, which is not always convenient.
Could you give us a real programming example on which getting access to response headers is not convenient? I think you must always check for response status if you are aiming at creating a robust application that deals with any kind of unexpected error which might happen (500, etc.)
De: <fiware-ngsi-bounces at lists.fi-ware.org<mailto:fiware-ngsi-bounces at lists.fi-ware.org>> on behalf of Tobias Jacobs <Tobias.Jacobs at neclab.eu<mailto:Tobias.Jacobs at neclab.eu>>
Fecha: martes, 7 de julio de 2015, 9:03
Para: FERMIN GALAN MARQUEZ <fermin.galanmarquez at telefonica.com<mailto:fermin.galanmarquez at telefonica.com>>
CC: "fiware-ngsi at lists.fi-ware.eu<mailto:fiware-ngsi at lists.fi-ware.eu>" <fiware-ngsi at lists.fi-ware.eu<mailto:fiware-ngsi at lists.fi-ware.eu>>
Asunto: Re: [Fiware-ngsi] FIWARE NGSI API discussion points
Dear Fermin and all,
due to time constraints [we have the IoT chapter F2F meeting these days] I have up to now only commented some of the discussion points. Additionally I have done the exercise to merge the different answers from the different emails, so [assuming I have not forgotten something] you can safely restrict your attention to this mail…
My new comments are written in green, more comments [and new discussion points] will follow.
Best
Tobias
[NEC]: For NEC it is important to preserve the possibility to have multiple instances of attribute values. This might not be so important for most of the cases where NGSI is used these days, where entities represent sensors and attributes represent their readings. However, in we expect that in IoT scenarios entities will represent arbitrary objects like rooms, houses, etc. and then there can indeed be several providers of the same attribute values who should be distinguishable.
However, we also see that this possibility should not make the life of users difficult who do not care about multiple attribute value instances because in their scenarios this does not play a role. Therefore our proposal is that NGSI systems treat attribute values with no given ID as having a specific “null” ID.
By this, the value ID possibility can be completely ignored by scenarios where it is not needed, while the NGSI standard still offers the possibility to have multiple values.
[FGM1] In order to fully understand you idea about IDs, could you elaborate an example aligned with the current NGSIv2 in order to see how it would looks like, please?
[TJ] Assume you have a context entity representing a street, and the co2-level of the street is measures by two different devices. In the NGSI model, the attribute “co2level” would then have two relevant values, each delivered by a distinct device, and these two values would be distinguished by distinct value IDs.
In FIWARE NGSI 1, applications who wish to retrieve only a specific attribute value could
- make a queryContextRequest and add to it an XPATH restriction which would limit the response to the attribute value instance having the desired value ID as metadata
- or use the convenience function making a GET on {ngsi10path}/contextEntities/myStreet/attributes/co2level/id1
A query not specifying the ID would, in contrast, return two attribute values.
In FIWARE NGSI 2 we can discuss whether the possibility to query a specific value needs to be included in the convenience function as a sub-resource of resources representing attributes – maybe this not necessary, and specific value instances can still be queried by using the corresponding restriction [restrictions are another question to discuss].
However, conceptually the NGSI system should distinguish between several value instances, so if two attribute values with different ID metadata exist, then e.g. an update on one instance should not overwrite the other.
--------------------------------
[TJ] The NGSI v2 proposal is very rich and there will be many different things to discuss or clarify. I am not sure what is the best “format” for the discussion, but let me start in this email with a few points (most of them questions). We are not done yet analyzing the complete proposal and more discussions will follow.
[FGM1] Email discussions are perfectly ok. Other alternative ways to provide feedback would be issues in github.com and even pull request suggesting modification to the specification or cookbook .apib files (.apib files at https://github.com/telefonicaid/fiware-orion/tree/develop/doc/apiary/v2 are the "source code" for Apiary web at http://telefonicaid.github.io/fiware-orion/api/v2/ and http://telefonicaid.github.io/fiware-orion/api/v2/cookbook)
(If you are not familiar with the pull request way of working in github.com, please tell me and we try to provide an introduction).
[TJ] OK; I think I prefer at the moment the email format for the basic discussions, and when it comes to the subtle details we can consider moving to pull requests. [Personally I am also not so well aware yet on how this works, but I have colleagues who can help me :)]
-----------------------------------
1) “flat” JSON message:
In general NEC agrees to reserve “id”, “type”, “value” to have special semantics. Even if this means that entities can have no attributes called “id”, “type” or “value”
Alternatively (and for maybe making the descriptions easier to understand) we might think about calling “id”, “type”, “value” special attributes with special semantics; some being mandatory (e.g. “id” is mandatory in entities).
-----------------------------------
2) Predefined value types: in general NEC supports to have predefined types in order to improve interoperability
-------------------------------------------------------------------
3) JSON types as predefined value types: We understand that a JSON property whose value is not a JSON object (and thus has no explicitly stated type) is implicitly either of type “number”, “string” or “boolean”, depending on the type of the JSON property. So for example "temperature": 32.5 would be implicitly of type “number”.
[FGM2] Yes, that's the idea. I would recommend you to have a new look to the specification, as we have merged some changes this morning that (we hope) make this idea clearer.
(The diff of the newest changes is at https://github.com/telefonicaid/fiware-orion/pull/1042/files in the case you prefer to see only the modifications)
[TJ] It seems in the newest version that you removed the “implicit” types number, boolean and string, so that every attribute value now needs to be a JSON object. Is this correct?
[FGM1] Not exactly... Every attribute needs to be a JSON type, which can be one of the following ones: string, number, boolean, object or array (i.e. not necessarily an "object", which is only one among 5 possibilities).
What I meant was that now an entity looking like
{
id: myStreet
co2Level: “aLot”
}
is not allowed, but needs to be something like
{
id: myStreet
co2Level: {
value: ”aLot”
type: “fun-string”
}
}
So the actual value is of JSON type string, but needs to be nested inside a JSON object where also the value type is declared.
At least this is how I interpret the “JSON entity representation” description in the ngsi2 draft, but maybe I still get it wrong?
-----------------------------------
4) Further predefined value types: We also understand that there will be further predefined types like for example “geo:point”. But we believe that attribute values having this kind of type will still need to be a JSON object with explicitly stated “type” property, because otherwise e.g. one cannot distinguish a “date” value from a “string” value. This would mean that the example "timestamp": "2017-06-17T07:21:24.238Z" is incorrect and would instead have to look like
“timestamp”:{
“value”: "2017-06-17T07:21:24.238Z"
“type”: date
}
[FGM3] In the new version of the spec, there are only two special NGSI types ("date" and "geo:point"), both associated to the JSON string type. I think it works in the way you suggest here, but a double checking the new version of the spec to get your feedback would be great.
[TJ] Yes, in the new versions the values are objects with explicitly given types.
[FGM2] Actually, values are JSON valid types, among string, number, boolean, object and array. For JSON strings, the attribute type (from a NGSI point of view) "date" and "geo:point" would have the special meaning described in the specification.
[TJ] see the example previous discussion point. What I meant is that now each attribute value is nested inside a JSON object…
-----------------------------------
5) Further predefined value types: We assume that some predefined value types can refer to specific JSON object formats, not only to specific string formats.
[FGM4] It could be so from a theoretical point of view. However, by the moment, the only two special types we have found make sense are "date" and "geo:point", both associated to JSON string. Maybe you have some other case in mind?
[TJ] I do not have a specific example in my mind at the moment, so it is indeed more a theoretical consideration. I imagine that for the sake of interoperability specific domains could publish something like a catalogue of their entity types and the format of the values they are using.
[FGM3] To be precise, I understand you mean "a catalogue of their entity types and the format of the values of the entity attributes they are using"
[TJ] Yes
[TJ] Probably we should first discuss what other predefined value types we want to have (if any), and only then discuss whether some of them should rather be JSON objects than specific string formats.
[FGM2] By the moment we have found "date" and "geo:point" and both fit into string JSON type, i.e. we haven't found the need to use a JSON obejct for them. Maybe some more examples arise as we progress in the NGSIv2 specification, but we don't foresee any new one at the current point of time.
-----------------------------------
6) User-defined types: It is currently written that “The values of user-defined informative types are represented by JSON string”, but we believe that user-defined types should be representable by numbers, Booleans and JSON objects as well.
[FGM5] This has changed in the new version of the spec and I guess now works in the way you suggest. Now, NGSI type and JSON type are "orthogonal" aspects on the attribute.
[TJ] OK, I read that even arrays would be allowed. Makes sense to me.
-----------------------------------
7) Error format: Currently an error code is not specified. Is this because it is intended to use the status code in the HTTP header instead? Can this lead to conflicts between standard HTTP error codes (e.g. 404 resource not found) and NGSI error codes (eg. 404 context element not found)?
[FGM6] Exactly. The idea is to use HTTP status code for the numeric error code, so no conflict should occur. Note that one of the criticized aspects in NGSI v1 is not using HTTP code to convey the error code, as this is not the common way of working in RESTful APIs. E.g. this would be a "not found" error response in NGSIv2:
HTTP/1.1 404 Not Found
Content-Length: 95
Content-Type: application/json
Date: ...
{
"error": "NotFound",
"description": "The requested entity has not been found. Check type and id"
}
-----------------------------------
[TJ] I need to further discuss with my colleagues about this before agreeing or proposing an alternative.
One concern is that not having the status code in the response message forces applications to also look into the response header, which is not always convenient.
[FGM4] (I understand that your refer to the status line, i.e. "HTTP/1.1 404 Not Found", not to response headers, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html)
On the other side, one of the concerns (that I have personally listened several times when presenting Orion in FIWARE events) is exactly the opposite :), i.e. that with the current way of working in NGSI you always need to check the payload to know the error. Even worse, depending on the request type (queryContext, updateContext, etc.) the error may be in different places on the payload, which is a little bit crazy from the point of view of the client programmer.
Anyway, the above approach should make both people happy, I mean:
* For people not wanting to check the status code, they can access to the payload and check the "error" message.
* For people not wanting to check the payload, they can get the status code in the status line.
8) API entry point
In general NEC agrees to make this resource somehow an overview of the main API urls. Would you make the format if this link list predefined, or would you leave the specific format to the API developer?
[FGM7] I think Jose Manuel could answer this better than me.
[JMCF] The API entry point must list at least all the URLs that allow to get access to the core functionalities offered by the API i.e.
Entities, types, subscriptions and registrations . In theory the only mandatory part would be the keys of the returned JSON object i.e.
entities_url, types_url, etc. As values yo could return your own URLs, but in practice it is very likely that you will be returning the same as in the example provided. I think this should be clarified in the spec.
-----------------------------------
9)
10)Pagination: In principle NEC supports the concept of pagination. However we are not sure how it can be realized in practice, as query results are dynamic (especially when getting the queried data live from a distributed system like IoT systems do). What the user probably expects is that the query results are “frozen” for him and then delivered page by page. But then something like a query ID is needed to collect the query results.
Alternatively the query result could be represented by a specific http resource (which is similar to a query id).
In both cases things become slightly more complex…
Do you have a proposal how to deal with that issue?
[FGM8] Frozen the query at NGSI server implementation could have an important performance impact (as the frozen results would need to be maintained in memory from an undefined amount of time while the client gets all the result) and would make NGSI sever implementation stateful, which is very bad from the point of view of HA and horizontal scaling.
Some ordering criteria may alleviate the problem In particular, entity creation date, which ensures that any new entity have to be put at the end. However, even in this case, there could be some problems if an entity already seen page is deleted (as it would introduce a shift in the next pages).
Thus, the solution based on limit/offset is not perfect, but may suffice most of the cases and have a good simplicity tradeoff. So, it is probably "the best of the worse" solution. But this point is still opened.
[TJ] I understand the scalability aspect.
On the other hand, NGSI systems who do not have the information ready in a database but need to assemble it from various sources anyway somehow have to cache the query result when it is retrieved page by page, as otherwise they would have to run the same queries on the subsystem again and again.
>From this point of view the question is whether clients should have separate control on data generation and on data retrieval, or if this difference should be transparent to applications. To be discussed.
[FGM5] I agree that pagination in a distributed context (as the architecture based on CB + CPrs is) is hard. We have been discussing about it at https://github.com/telefonicaid/fiware-orion/issues/847 long time, but without having found a perfect solution.
-----------------------------------
________________________________
Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-ngsi/attachments/20150707/c4179de2/attachment.html>
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy