[Fiware-ngsi] Regarding NGSIv2 - Strategic Concerns

[Tobias Jacobs]

Dear all,

Regarding the strategic concerns raised by Ernoe, we have to insist that the impacts of switching to NGSI 2 at this time should be broadly discussed within the consortium.
NGSI 1.0 is the de-facto standard implemented by many enablers, each of which will be impacted if FIWARE officially moves to NGSI 2.0. For sure it will not serve the project’s reputation if there is only a single GEri compliant with the final API standard.

Independently from these general concerns, we have prepared a first version of release notes to be potentially attached to NGSI 2.0, should it be decided to be released in its current state.

I insert it in text format below; please feel free to suggest a more suitable format to maintain these notes.

Best regards
Tobias



NGSI v2 Release Notes, May 2016

FIWARE NGSI v2 is emerging as a major evolution of the previous FIWARE NGSI 1.0 standard. The main objective of defining version 2 is to simplify representations and interactions, remove unused features, and introduce new features that have turned out to be useful.

These specifications of FIWARE NGSI v2 are to be regarded as preliminary, that is, they are subject to changes in the near future. In principle, the potential for future changes is not limited to specific sections but can affect any part of the document.

Nevertheless, the following list of unresolved issues can serve as a guide to understand which parts of the specification are more likely to change than others. Developers are particularly requested to notify which proposed changes will potentially affect their specific implementations.


1) Compatibility with JSON-LD representation format.
Since early 2016 discussions between FIWARE participants have started to take place about whether and how to use the JSON-LD format for NGSI messages, with the purpose that FIWARE NGSI becomes compatible to Linked Data enablers. A number of decisions on the representation have been agreed on (e.g. how to represent metadata), some questions are still under discussion (e.g. whether entity IDs should be dereferenceable), and some questions have not been addressed yet (e.g. how to represent context registrations in JSON-LD).

There is no agreement yet on whether the JSON-LD format has to be aligned with the JSON format used in the current NGSI 2 specifications. If the decision for alignment is taken affirmatively, this will have potential impacts on the whole JSON representation format of NGSI 2. In particular, an impact on the following aspects is foreseen:

1a) Representation of Metadata

1b) Alternative entity representations (normalized, keyValues mode,values mode)

1c) Required possibility to use full URIs to represent entity IDs, entity types, value types.

1d) Definition of URIs for predefined attributes and types.


2)        Concepts and features proposed from NGSI 1.0 proposed to be removed

A number of concepts in NGSI 1.0 have been removed in the current NGSI 2 document, but for many of these changes there is no agreement yet among the FIWARE members.

2a) Expressiveness of restriction language.
In NGSI 1.0 with xml serialization, XPATH expression are foreseen be used to filter query responses. Current NGSI 2.0 specifications use a different restriction format. The consequences of this and possible problems have not yet been discussed.

2b) Operation Scopes
Scopes are defined in NGSI 1.0 as a means to restrict the reach of operations. This is relevant in particular for large federated systems, where scopes offer the possibility to define on which parts of the system the query should be executed. In the current NGSI 2 specifications there is no concept of scopes.

2c) Multiple instances of attribute values.
In NGSI 1.0 a system can maintain multiple values of the same attribute of the same entity, and these values can be independently created, updated, and deleted. In contrast, the current version of NGSI 2 is based on the assumption that every attribute only has a single value. Resolving this issue will have potential impacts on (i) attribute value representation, (ii) syntax and semantics of update operations, (iii) a predefined attribute value id metadata like defined in NGSI 1,0.

2d) Distinction between Context Elements and Context Entities.
In NGSI 1.0 there is a formal distinction between Context Entities (representing physical objects), and Context Elements (representing containers for information about the physical objects). Regarding NGSI 2, this distinction is still under discussion. The distinction has also be proposed in the JSON-LD (see 1)) discussions as a potential solution to the technical problem of dereferenceability.

2e) Possibility to specify metadata for each individual entity and/or attribute in registrations.

3)        Features introduced in NGSI 2

In the current version of NGSI 2 a number of new features are introduced that have not been agreed on yet. For some of these features the concern was raised that they make implicit assumptions on the architecture of the NGSI system (assuming a centralized architecture), because with other architectures (federated) these features cannot be efficiently realized.

3a) Pagination of responses

3b) Behavior of "types" resource and sub-resources. In the current NGSI 2 document the number of entities of each entity type can be retrieved. For NGSI systems this might not be desirable or not even possible to return, which is typical for federated systems, but occurs also however in systems with (a) rapid dynamic changes (b) authorization restrictions.

3c) "servicePath" feature


4)        Completeness of Specifications
A number of incomplete specifications in the current NGSI 2 descriptions have been identified and need further discussion to avoid misunderstandings and/or incompatible implementations.

4a) Context Availability.
Much of the functionalities regarding context availability information (context registrations) present in FIWARE NGSI 1.0 is not yet covered by the NGSI 2.0 specifications. This includes in particular context availability subscriptions and notifications.

4b) Role of Components.
It has been discussed several times to define roles of components, and for each role define the set of required functionalities. In the current specifications the only role referred to is "NGSI server", which is not clearly defined. It has been requested that (i) Context Availabilty and Context Data are separated by different roles, and (ii) that there is a role for lightweight NGSI components with reduced set of features.


4c) Clarification on system assumptions
FIWARE NGSI 1.0 is a pure interface specification and does not make any assumption about the system implementing it. In particular, it does not assume any system state model. For example, there is no requirement in FIWARE NGSI 1.0 that an update operation on an attribute value of an entity will effectuate that the same attribute value is returned as a query response.
In other words, any component accepting syntactically correct NGSI 1.0 messages and returning syntactically correct responses as describd in the interface specifications is compliant with FIWARE NGSI 1.0.
With regard to NGSI 2.0 there has been no discussion yet about this question, and parts of the current specifications read like it is assumped that the NGSI system maintains the context information in a database. For example, it is mentioned several times that entities are "created".


4d) Error Handling.
There are missing formal specification of error messages and their contents for each operation. For example, writing about the "affectedItems" field of error messages "Depending on the operation, it may refer to entities, registrations or subscriptions" is not sufficient to understand the syntactical requirements for each particular operation.

4e) Clarifications regarding optionality
A clear distinction between required and optional features is required. For example, from the formulation "the symbol ':' can be used as a synonym for '=='" it does not become clear whether NGSI clients are free to use either (and therefore the API must support both), or whether it is the compliant NGSI server who can decide which syntax to support.

4f) Table of changes from NGSI 1.0 to 2.0.
An overview of differences between NGSI 1.0 and  version 2 should be given, preferably with a summary table.


5) Specification Style and Clarity of Normative Text

It is under discussion whether the current NGSI 2 descriptions satisfy the requirement for the formal specification of a standard. The following gaps are under consideration.

5a) The use of key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” must be used and be consistent with RFC2119.

5b) URL Parameters.
The syntax and semantics of the URL parameter specification language are not formally defined. In particular, it is not unambiguously clear which combinations of parameters are acceptable for which request (this impacts Error Handling). Furthermore, the semantics of the geographical query language is not clearly defined, e.g. it does not become clear how the distance between a point and a polygon is defined (obviously leading to inconsistent returned search results).

5c) Normative text.
There is currently no distinction between normative specification and informative examples.

5d) Clarity of normative text.
Missing clarity in descriptions. One of many examples is the sentence "this mode is just like values mode, with the only difference that values are not repeated".

5e)
Tutorial-style language, like e.g. "You can use a simple quote", or "Let's consider the following ..."

5f)
Order of descriptions prevents a good reading flow. For example, in the "virtual attributes" section references are made to details of the URI parameter language, which at this point has not yet been introduced.

5g)
Inconsistencies, e.g. ordering of results by attribute values is defined, but for attribute values which are JSON objects or lists the semantics of the ordering is not clear.

5h)
Comments that do not belong into a specification document , like "it is noteworthy the fact that spatial properties need special indexes which can be under resource constraints imposed by backend databases"

5i)
Inappropriate language style like "it only makes sense when used with attributes which contain dates". In a specification the question is not whether something makes sense, but whether it is syntactically correct and what are the semantics.

5j)
missing formal specification of syntax and semantics of the notification customization language. It is a "specification by example" in the current NGSI 2 descriptions.






From: fiware-ngsi-bounces at lists.fiware.org [mailto:fiware-ngsi-bounces at lists.fiware.org] On Behalf Of Fermín Galán Márquez
Sent: Donnerstag, 26. Mai 2016 09:28
To: Ernoe Kovacs
Cc: fiware-ngsi at lists.fiware.org
Subject: Re: [Fiware-ngsi] Regarding NGSIv2 - Strategic Concerns


Dear Enroe,

I think there isn't any risk of leaving the platform in disorders a long as NGSIv2 implemenations would be done in a incremental way, without discontinuing NGSIv1. At least, that is the approach we are adopting in Orion Context Broker GEri which currently support NGSIv1 and, at the same time, a version of NGSIv2 pretty much aligned with what would be released as candidate release very soon.

Best regards,

------

Fermín

PD. I'm not a member of Foundations or Architecture Boards, so feel free to give the weight you want to my answer ;)

El 25/05/2016 a las 9:21, Ernoe Kovacs escribió:
Dear NGSI community,

Following up on our discussion in the telco last Wednesday, NEC confirms to support adaption of NGSI into a more attractive version 2,
because it is vital to the success of a scalable non-proprietary FIWARE global platform.

Regarding our technical concerns with the current draft, we support creation of specs and Release Notes for a NGSIv2, as a necessary tool
for active co-development within the FIWARE community. NEC commits to providing text/issues/use-cases/proposed-solutions for the Release Notes.

NEC additionally requests that the following strategic concerns be answered, presumably by decisions at Foundation (or Architecture Board) level:
a) Which components/enablers will be affected by the proposed v2 changes?
b) Do the relevant Partners agree to change the code aspects needed
     by their components to show interworking with the proposed NGSIv2
     within the lifetime of the current project (about 3 months) ?
     If not, how will FIWARE succeed at the Final Review if some partners have
     changed and others have not, leaving the platform in disorder?
     Alternatively, will FIWARE Partners apply for an extension of time (same funding) ?
c) If Partners agree in principle within (b), when will the decision be ratified at the
     level of the Architecture Board (or higher)?

Until final decision of FIWARE in  above, the specifications and Release Notes should be labelled "Candidate Release v2"

Kind regards

-        Ernoe & the NEC team

________________________________

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-ngsi/attachments/20160527/c9049c7d/attachment.html>