Hello,
Here is some additional remarks (in red), most of which were discussed
at the F2F meeting.
I also put some links to discussions that already took place in the
issue tracker when I found similar remarks.
Marc
----
[TJ] could you list the most important points in terms of functionality
that are added / removed by your proposal? E.g. I think that the
attribute domains have been thrown out; at least I do not see them
currently.
[FGM] Added:
o Geolocation and filtering capabilities (actually, I'd say that
they aren't addition to NGSI itself -as their implemention in
the standard operations family relies on existing
Scope/Restriction structures-, but an addition in the way of
using them for filtering).
o Pagination (already in Orion API for v1)
o Get entity types
o Get subscription list
o Get registration list
o Get subscription availability list
* Removed:
o Special semantics for metadata ID, i.e. ID is now a regular
metadata (nobody was using these special semantcis, see my email
to this list on June 2nd, 2015)
o Attribute domains (you can get the same using attributes which
value is a key-map).
[TJ] One important point would be the restriction language. We used to
have XPATH expressions for the XML message bodies, but they do not seem
to have been used so much, and we talk about json messages now.
In the current ngsi2 proposal it seems restrictions on attribute values
are supported as URL parameters, but filtering based on metadata (e.g.
for retrieving only attribute values received from a certain provider)
is not possible by this, right?
Have you thought about the language for restrictions? Something as
powerful/complex as the previous xpath?
[FGM] Yes, we have considered the possibility of allowing a more
powerfull query language (in addition to the simple one used by the "q"
argument, that may suffice most of use cases) and even considering some
alternatives, e.g. MQL. A complex language doesn't seems to be a good
idea to be included in a URL param, so maybe makes more sense using query.
Please have a look to
https://github.com/telefonicaid/fiware-orion/issues/1051. Jose Manuel is
leading this part, so probably he may add additional comments about the
status as it evolves.
-------------
[TJ] Do you intend to keep the distinction between NGSI 9 and NGSI 10?
In my view it would be useful to at least sharply distinguish between
operations for exchanging information and operations for exchanging
“availability information”.
[FGM] That difference could be explained in the introductory sections,
before depicting each operation. What we have found is that people find
confusion "NGIS9" and "NGSI10" (they tend to confuse with version 9 and
version 10 of a thing called "NGSI"), so probably we should discuss
about "context" and "context availability" but without mentioning NGSI9
and NGSI10.
[JH] I rather believe that distinguishing between NGSI9 and NGSI10
doesn't make so much sense from a developer's perspective.
[TJ] NEC agrees that we could get rid of the “API version numbers”
NGSI-9 and NGSI-10.
But from the IoT Chapter point of view there should be a distinction
between context-data and context-availability. In the IoT Backend the
functionality is split between a component maintaining the availability
information (“IoT Discovery”) and data (“IoT Broker”).
This sharp split e.g. also better enables distributed setups where a
central IoT Discovery component provides several instances of IoT Broker
with availability information.
Concretely, what we propose is to have context data (former NGSI-10) and
context availability (former NGSI-9) on different resource trees like we
had it previously; e.g. {serverRoot}/contextData and
{serverRoot}/contextAvailability .
-------------
- [TJ] I think it would be good to keep the approach of having a
set of basic operations (queryContextRequest etc.) with which in
principle one can do everything, so that convenience operations only add
convenience but not add functionality as compared to the basic operations?
[FGM] In general, I agree. In fact, we plan to include standards
operations also as part of the reference documentation. As I mention in
my previous email: /"The current draft at the above URL is focused on
not-standard operations (sometime referred as "convenience operations")
but at the end it will also include a specification for the standard
operations (e.g. queryContext, updateContext, registerContext, etc.)
based in the one we already have for the JSON encoding of that
operations in the v1 API"./ However, there are some exceptions, eg: the
convenience operation to get all the entity types ("GET /v2/types") in
the sytesm doesn't have any counterpart in the standards operation
family. But I understand that this is not a mayor problem.
[JH] We definitively should keep the non-convenience operations (I
wouldn't call them "standards" by the way, maybe we need to call them in
another way). However, I understand that we will implement some of the
simplifications in the returned JSON structures, aren't we? Leaving
the developer the option to choose between self-contained (verbose :-)
JSON structures and more simplified structures also would apply for
these kind of operations.
[MC] As newcomers to the NGSI / Fiware ecosystem, me and some colleagues
of mine have found very confusing that NGSI v1 exposes multiple ways of
accessing the exact same information thought the standard and
convenience operations. After a quite heated discussion with Tobias at
the F2F meeting in Heidelberg, I now have a better understanding of the
historic reasons that pushed the decision to have a subset of the NGSI
standard operations duplicated as "convenience" REST endpoints... but I
still think it was an error in the first place. Accepting that some
developers need a more "convenient" way to access data is accepting that
standard operation are not convenient (by not being RESTfull) and that
design of the NGSI v1 APIs is flawed in some way.
The first thing to decide (and there was no consensus at the F2F
meeting) is whether NGSI v2 is about :
- breaking compatibility with v1 allowing to change all the APIs,
- keeping compatibility with v1 and just adding missing functionalities
(more a v1.1),
We need to decide on that first.
Then, *if* the NGSI v2 specification is about breaking compatibility
with v1 and simplifying it using REST endpoints, it *may* be the right
time to discuss if the standard operations need to be included in v2 and
if we can move all the NGSI functionalities to "REST-like" endpoints. I
say only "REST-like" as some NGSI concepts cannot be mapped fully to
REST properly.
On another side, I don't have anything against keeping the standard
operations for v2. But then, we should accept that the NGSI model is not
adapted to the REST conventions and don't try to apply REST partially by
duplicating endpoints.
[FGM] NGSIv2 is a major rework of the FIWARE NGSI binding, so we can
assume breaking compatibility with v1. From a client perspective, there
should be no problem as long as the endpoint for v1 and v2 are
separated, e.g. in Orion v1 uses <host>:<port>/v1 and v2 uses
<host>:<port>/v2. In this context, keeping or not keeping standard
operations is an open point. I think that they could be kept as long as
they provide functionality not covered by non-standard operations, e.g.
standard queryContext can query by a list of entities thus is a kind of
bulk/batch query operation that make sense to be included. Probably we
need to examine standard operation by standard operation to decide which
ones are useful and which ones aren't.
By the moment, in seems that queryContext make sense (not yet included
in http://telefonicaid.github.io/fiware-orion/api/v2/, PR pending
https://github.com/telefonicaid/fiware-orion/pull/1060)
-------------
[NEC]: For NEC it is important to preserve the possibility to have
multiple instances of attribute values. This might not be so important
for most of the cases where NGSI is used these days, where entities
represent sensors and attributes represent their readings. However, in
we expect that in IoT scenarios entities will represent arbitrary
objects like rooms, houses, etc. and then there can indeed be several
providers of the same attribute values who should be distinguishable.
However, we also see that this possibility should not make the life of
users difficult who do not care about multiple attribute value instances
because in their scenarios this does not play a role. Therefore our
proposal is that NGSI systems treat attribute values with no given ID as
having a specific “null” ID.
By this, the value ID possibility can be completely ignored by scenarios
where it is not needed, while the NGSI standard still offers the
possibility to have multiple values.
[FGM] In order to fully understand you idea about IDs, could you
elaborate an example aligned with the current NGSIv2 in order to see how
it would looks like, please?
[TJ] Assume you have a context entity representing a street, and the
co2-level of the street is measures by two different devices. In the
NGSI model, the attribute “co2level” would then have two relevant
values, each delivered by a distinct device, and these two values would
be distinguished by distinct value IDs.
In FIWARE NGSI 1, applications who wish to retrieve only a specific
attribute value could
-make a queryContextRequest and add to it an XPATH restriction which
would limit the response to the attribute value instance having the
desired value ID as metadata
-or use the convenience function making a GET on
{ngsi10path}/contextEntities/myStreet/attributes/co2level/id1
A query not specifying the ID would, in contrast, return two attribute
values.
In FIWARE NGSI 2 we can discuss whether the possibility to query a
specific value needs to be included in the convenience function as a
sub-resource of resources representing attributes – maybe this not
necessary, and specific value instances can still be queried by using
the corresponding restriction [restrictions are another question to
discuss].
However, conceptually the NGSI system should distinguish between several
value instances, so if two attribute values with different ID metadata
exist, then e.g. an update on one instance should not overwrite the other.
[JMCF] I’m sympathetic to what you are proposing but I was wondering if
what you are pursuing might result in API over engineering just to
support a use case that can be seamlessly handled by the application
itself. For instance, you can have a CO2 level value which can be
obtained from the contribution of several measurements coming from
different sensors . You might even automatically update the resulting
value when a new measurement arrives by using the CEP. For instance.
{
id: “Friedrichstrasse”,
type: “Street”,
co2Level: 0.1
}
{
id: “Sensor-789”,
type: “CO2Sensor”,
location: “ Friedrichstrasse”,
measurement: 0.15
}
{
id: “Sensor-900”,
type: “CO2Sensor”,
location: “ Friedrichstrasse”,
measurement: 0..09
}
[TJ]: In your example you refer to entities representing the different
sensors, and of course such a representation is always possible.
In the IoT chapter we typically speak of “higher-level” entities
representing more abstract Things like streets, and we expect that the
IoT user is not aware of which are the names of the sensors who deliver
the information., so the user would query for “street” instead of
“Sensor-900”. And it is in such scenarios where the same attribute is
potentially delivered by several providers.
[FGM] In an scenario in which you combine low level and high level
entities, you can always ask to low level entities for the particular
values is you need. Moreover, the advantage of having the high level
entity is to hide the details of the low level entities (i.e.
abstraction), thus is you consider the posibility to ask for low level
entities /through the high level entity/ (using IDs or other means) you
are breaking that abstraction.
--------------------------------
[TJ] The NGSI v2 proposal is very rich and there will be many different
things to discuss or clarify. I am not sure what is the best “format”
for the discussion, but let me start in this email with a few points
(most of them questions). We are not done yet analyzing the complete
proposal and more discussions will follow.
[FGM] Email discussions are perfectly ok. Other alternative ways to
provide feedback would be issues in github.com and even pull request
suggesting modification to the specification or cookbook .apib files
(.apib files at
https://github.com/telefonicaid/fiware-orion/tree/develop/doc/apiary/v2
are the "source code" for Apiary web at
http://telefonicaid.github.io/fiware-orion/api/v2/
<http://telefonicaid.github.io/fiware-orion/api/v2/> and
http://telefonicaid.github.io/fiware-orion/api/v2/cookbook
<http://telefonicaid.github.io/fiware-orion/api/v2/cookbook>)
(If you are not familiar with the pull request way of working in
github.com, please tell me and we try to provide an introduction).
[TJ] OK; I think I prefer at the moment the email format for the basic
discussions, and when it comes to the subtle details we can consider
moving to pull requests. [Personally I am also not so well aware yet on
how this works, but I have colleagues who can help me J]
-----------------------------------
1)“flat” JSON message:
In general NEC agrees to reserve “id”, “type”, “value” to have special
semantics. Even if this means that entities can have no attributes
called “id”, “type” or “value”
Alternatively (and for maybe making the descriptions easier to
understand) we might think about calling “id”, “type”, “value” special
attributes with special semantics; some being mandatory (e.g. “id” is
mandatory in entities).
-----------------------------------
2)Predefined value types: in general NEC supports to have predefined
types in order to improve interoperability
-------------------------------------------------------------------
3)JSON types as predefined value types: We understand that a JSON
property whose value is not a JSON object (and thus has no explicitly
stated type) is implicitly either of type “number”, “string” or
“boolean”, depending on the type of the JSON property. So for example
"temperature": 32.5 would be implicitly of type “number”.
[FGM] Yes, that's the idea. I would recommend you to have a new look to
the specification, as we have merged some changes this morning that (we
hope) make this idea clearer.
(The diff of the newest changes is at
<https://github.com/telefonicaid/fiware-orion/pull/1042/files>https://github.com/telefonicaid/fiware-orion/pull/1042/files
in the case you prefer to see only the modifications)
[TJ] It seems in the newest version that you removed the “implicit”
types number, boolean and string, so that every attribute value now
needs to be a JSON object. Is this correct?
[FGM] Not exactly... Every attribute needs to be a JSON type, which can
be one of the following ones: string, number, boolean, object or array
(i.e. not necessarily an "object", which is only one among 5 possibilities).
What I meant was that now an entity looking like
{
id: myStreet
co2Level: “aLot”
}
is not allowed, but needs to be something like
{
id: myStreet
co2Level: {
value: ”aLot”
type: “fun-string”
}
}
So the actual value is of JSON type string, but needs to be nested
inside a JSON object where also the value type is declared.
At least this is how I interpret the “JSON entity representation”
description in the ngsi2 draft, but maybe I still get it wrong?
[JMCF]
{
id: myStreet
co2Level: “aLot”
}
In this variant there is no NGSI type informed. In the variant which
include type you are informing a NGSI type but from the point of view of
the implementation they are both strings.
[TJ] OK, then my question would probably be whether an NGSI value type
is mandated for attribute values. From reading the current specs my
guess would be “yes”, but I would like to have it confirmed…
[FGM] NGSI type for attributes is not mandatory in NGSIv2 (in fact, it
is not mandatory in OMA NGSI, see Section 5.5.3), for example the
fragment above from JMCF.
-----------------------------------
4)Further predefined value types: We also understand that there will be
further predefined types like for example “geo:point”. But we believe
that attribute values having this kind of type will still need to be a
JSON object with explicitly stated “type” property, because otherwise
e.g. one cannot distinguish a “date” value from a “string” value. This
would mean that the example /"timestamp": "2017-06-17T07:21:24.238Z" /is
incorrect and would instead have to look like
/“timestamp”:{
“value”: "2017-06-17T07:21:24.238Z"
“type”: date
}/
[FGM] In the new version of the spec, there are only two special NGSI
types ("date" and "geo:point"), both associated to the JSON string type.
I think it works in the way you suggest here, but a double checking the
new version of the spec to get your feedback would be great.
[TJ] Yes, in the new versions the values are objects with explicitly
given types.
[FGM] Actually, values are JSON valid types, among string, number,
boolean, object and array. For JSON strings, the attribute type (from a
NGSI point of view) "date" and "geo:point" would have the special
meaning described in the specification.
[TJ] see the example previous discussion point. What I meant is that now
each attribute value is nested inside a JSON object…
[FGM] Not exaclty... actually, there are two possibilities. Citing
from current NGSIv2 spec draft: /"Properties representing attributes
with no metadata or type are rendered / /_as regular JSON values_//[i.e.
not nested] Properties representing attributes which specify a type or
include metadata are rendered //_as JSON objects_//[i.e. netsted]"./
-----------------------------------
5)Further predefined value types: We assume that some predefined value
types can refer to specific JSON object formats, not only to specific
string formats.
[FGM] It could be so from a theoretical point of view. However, by the
moment, the only two special types we have found make sense are "date"
and "geo:point", both associated to JSON string. Maybe you have some
other case in mind?
[TJ] I do not have a specific example in my mind at the moment, so it is
indeed more a theoretical consideration. I imagine that for the sake of
interoperability specific domains could publish something like a
catalogue of their entity types and the format of the values they are using.
[FGM] To be precise, I understand you mean "a catalogue of their entity
types and the format of the values _of the entity attributes_ they are
using"
[TJ] Yes
[TJ] Probably we should first discuss what other predefined value types
we want to have (if any), and only then discuss whether some of them
should rather be JSON objects than specific string formats.
[FGM] By the moment we have found "date" and "geo:point" and both fit
into string JSON type, i.e. we haven't found the need to use a JSON
obejct for them. Maybe some more examples arise as we progress in the
NGSIv2 specification, but we don't foresee any new one at the current
point of time.
-----------------------------------
6)User-defined types: It is currently written that /“The values of
user-defined informative types are represented by JSON string”/, but we
believe that user-defined types should be representable by numbers,
Booleans and JSON objects as well.
[FGM] This has changed in the new version of the spec and I guess now
works in the way you suggest. Now, NGSI type and JSON type are
"orthogonal" aspects on the attribute.
[TJ] OK, I read that even arrays would be allowed. Makes sense to me.
-----------------------------------
7)Error format: Currently an error code is not specified. Is this
because it is intended to use the status code in the HTTP header
instead? Can this lead to conflicts between standard HTTP error codes
(e.g. 404 resource not found) and NGSI error codes (eg. 404 context
element not found)?
[FGM] Exactly. The idea is to use HTTP status code for the numeric error
code, so no conflict should occur. Note that one of the criticized
aspects in NGSI v1 is not using HTTP code to convey the error code, as
this is not the common way of working in RESTful APIs. E.g. this would
be a "not found" error response in NGSIv2:
HTTP/1.1 404 Not Found
Content-Length: 95
Content-Type: application/json
Date: ...
{
"error": "NotFound",
"description": "The requested entity has not been found. Check type
and id"
}
-----------------------------------
[TJ] I need to further discuss with my colleagues about this before
agreeing or proposing an alternative.
One concern is that not having the status code in the response message
forces applications to also look into the response header, which is not
always convenient.
[FGM] (I understand that your refer to the status line, i.e. "HTTP/1.1
404 Not Found", not to response headers, see
http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html)
On the other side, one of the concerns (that I have personally listened
several times when presenting Orion in FIWARE events) is exactly the
opposite :), i.e. that with the current way of working in NGSI you
always need to check the payload to know the error. Even worse,
depending on the request type (queryContext, updateContext, etc.) the
error may be in different places on the payload, which is a little bit
crazy from the point of view of the client programmer.
Anyway, the above approach should make both people happy, I mean:
?For people not wanting to check the status code, they can access to the
payload and check the "error" message.
?For people not wanting to check the payload, they can get the status
code in the status line.
[JMCF] Could you give us a real programming example on which getting
access to response headers is not convenient? I think you must always
check for response status if you are aiming at creating a robust
application that deals with any kind of unexpected error which might
happen (500, etc.)
------------------------------------------------------------
8)API entry point
In general NEC agrees to make this resource somehow an overview of the
main API urls. Would you make the format if this link list predefined,
or would you leave the specific format to the API developer?
[FGM] I think Jose Manuel could answer this better than me.
[JMCF] The API entry point must list at least all the URLs that allow to
get access to the core functionalities offered by the API i.e.
Entities, types, subscriptions and registrations . In theory the only
mandatory part would be the keys of the returned JSON object i.e.
entities_url, types_url, etc. As values yo could return your own URLs,
but in practice it is very likely that you will be returning the same as
in the example provided. I think this should be clarified in the spec.
-----------------------------------
10)Pagination: In principle NEC supports the concept of pagination.
However we are not sure how it can be realized in practice, as query
results are dynamic (especially when getting the queried data live from
a distributed system like IoT systems do). What the user probably
expects is that the query results are “frozen” for him and then
delivered page by page. But then something like a query ID is needed to
collect the query results.
Alternatively the query result could be represented by a specific http
resource (which is similar to a query id).
In both cases things become slightly more complex…
Do you have a proposal how to deal with that issue?
[FGM] Frozen the query at NGSI server implementation could have an
important performance impact (as the frozen results would need to be
maintained in memory from an undefined amount of time while the client
gets all the result) and would make NGSI sever implementation stateful,
which is very bad from the point of view of HA and horizontal scaling.
Some ordering criteria may alleviate the problem In particular, entity
creation date, which ensures that any new entity have to be put at the
end. However, even in this case, there could be some problems if an
entity already seen page is deleted (as it would introduce a shift in
the next pages).
Thus, the solution based on limit/offset is not perfect, but may suffice
most of the cases and have a good simplicity tradeoff. So, it is
probably "the best of the worse" solution. But this point is still opened.
[TJ] I understand the scalability aspect.
On the other hand, NGSI systems who do not have the information ready in
a database but need to assemble it from various sources anyway somehow
have to cache the query result when it is retrieved page by page, as
otherwise they would have to run the same queries on the subsystem again
and again.
From this point of view the question is whether clients should have
separate control on data generation and on data retrieval, or if this
difference should be transparent to applications. To be discussed.
[FGM] I agree that pagination in a distributed context (as the
architecture based on CB + CPrs is) is hard. We have been discussing
about it at https://github.com/telefonicaid/fiware-orion/issues/847 long
time, but without having found a perfect solution.
---
[MC] Query expressions of the "q" parameter only applies on attributes
and not metadata. We may need a syntax to tell the matching should occur
on metadata of a given attribute like "temperature/unit" to match the
unit metadata field of a temperature attribute.
---
[MC] There would be 3 formats for attributes, as simple JSON values, as
JSON object (for type and metadata) and canonical. Maybe only one could
be standardized, others would be implementation specific (only supported
in Orion CB for third party clients) ?
---
[MC] For registrations/subscriptions, how do we define if we want to
match a complete id or an idpattern in the subject/entity field ?
---
[MC] Orion CB supports Service/ServicePath HTTP headers. Should they be
defined in the spec ? Is it a requirement for other GE to implement for
interoperability ?
---
[MC] Mixing of structured attributes values could be a problematic to
handle (already discussed on the issue tracker:
https://github.com/telefonicaid/fiware-orion/issues/1040).
---
[MC] Shoud registration/subscription support the DELETE verb, or are we
still relying on duration set to 0 ?
---
[MC] Does registration (like subscription) support multiple callbacks
(providingApplication) for the same subject ? Would it allow fanout for
query/updates of (IoT Agent) lazy/commands attributes ?
---
[MC] Support for JSON-LD was discussed during the F2F meeting : JSON-LD
@id, @type could simply be stored as normal attributes. But an
interesting solution is proposed in the tracker
(https://github.com/telefonicaid/fiware-orion/issues/1053)
---
[MC] The specification is missing some NGSI definitions like Service
Description and Information Model part of the v1 specification.
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-ngsi/attachments/20150710/91438164/attachment.html>
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy