[Fiware-testbed] Observationsdraft for 10.5.1 report on validation process

[Sandfuchs, Thorsten]

Status: I got "one" reply on the questions raised.

Clarification from the WPL/WPA meeting:

-          The observation letter will be send out on a work package level, as soon as we have a consolidated letter.

-          This particular observation letter should be forwarded to the WPL/WPA list for additional comments.

In the PhC today we have to decide if we still need changes and when we send the letter for further processing.

Best,
                                                                /Thorsten


From: Sandfuchs, Thorsten
Sent: Mittwoch, 24. April 2013 23:26
To: Sandfuchs, Thorsten; stefano de panfilis; fiware-testbed at lists.fi-ware.eu; JUAN JOSE HIERRO SUREDA
Subject: RE: [Fiware-testbed] Observationsdraft for 10.5.1 report on validation process

Dear colleages,
The  only comment I got was the new yellow sentence down below. Anybody else?

Could you quickly confirm if you

[] took a look at it and found it reasonable and OK

[] took a look at it and found it not reasonable and not OK

[] if you won't be able to read it for the next 2 weeks (by then it needs to be long send out)

[] if you want to read it and this should block further processing

[] if you want to read it and this should not block further processing

Next step would be to present this to the WPL/WPA of the other chapters.

Thanks for any support.


                                                /Thorsten

From: fiware-testbed-bounces at lists.fi-ware.eu<mailto:fiware-testbed-bounces at lists.fi-ware.eu> [mailto:fiware-testbed-bounces at lists.fi-ware.eu] On Behalf Of Sandfuchs, Thorsten
Sent: Montag, 22. April 2013 20:41
To: stefano de panfilis; fiware-testbed at lists.fi-ware.eu<mailto:fiware-testbed at lists.fi-ware.eu>; JUAN JOSE HIERRO SUREDA
Subject: [Fiware-testbed] Observationsdraft for 10.5.1 report on validation process


Dear colleagues,

Following up on the review report, find down below my draft of four observations given to the review report conserning the rejection of D.10.5.1



TID clarified with the Project Officer: The EC will not put into question the assessment of suitably qualified experts. Yes-no discussions about opinions are not very meaningful, and we would not enter into such discussions. Only factual statements (and not opinions dressed up as factual statements, as is often the case) would be considered



Please excuse the "german" style, potentially having to many lengthy sentences down below and feel free to break things up or change them at your will :)



Please provide your input and corrections as soon as possible. In order to join the discussion and to cater for a shorter "release"-cycle I will add the chief architect directly to this discussion.



Best,

                                               /Thorsten


PS: find the original reviewer comments at the bottom of this mail


Observation towards "D10.5.1 Report on Validation Process including Validation with Use Case projects" - Deliverable D10.5.1 is rejected. No re-submission is required.


1.       "The validation process described in the document is generally well thought and detailed; however, it has been devised without sufficient consideration of the FI-WARE project and FI-WARE Releases."

Observation by FI-WARE:

-          The validation process was initially proposed by FI-WARE to the Use-Case projects. This approach was discussed and agreed by representatives of the Use-Case projects within the Architecture Board.

-          The architecture board recognized the reviewers comments in M6 and FI-WARE proposed a subsequent "initial feedback survey" to cater for short-term feedback cycle and to allow feedback on the first Testbed instances (therefore being closely aligned with the FI-WARE project and FI-WARE release - major fi-ware release end of M17/30.09.2012 - survey send out: Di 02.10.2012 01:03).

-          Subsequent and proposed validation schedules are by definition of the process to be aligned within the involved stakeholders: FI-WARE and use case projects. There is no defined schedule by the validation process, therefore alignment towards subsequent FI-WARE Release is part of the involved partners and not of the process. The process allows maximum flexibility.


2.       "The validation approach is also considered insufficient, in view of what is envisaged in the DoW in supporting Use Case projects on deployment, execution and validation of the conceptual prototypes in respect of the available GEs. "

Observation by FI-WARE:

-          FI-WARE agrees with the reviewers comments that the validation process as outlined and executed within the FI-WARE project within Task 10.5 does not completely follow the description of the task in the DoW - but again as this was decided and discussed within the highest possible technology board (the FI-WARE architecture board) this deviation from the DoW is in line with all related parties and the deviation in general should be acceptable. As changes to the DoW take time, these changes in the process could only be reflected as part of the upcoming amendment 4 to the DoW.


3.       "According to the deliverable, the design phase of FI-WARE incorporates requirements that have been successfully communicated from the Use Cases Projects to the FI-WARE chapters. As the link between Use Case requirements and the actual content of the individual chapters is not readily traceable, this has a significant impact on the validation, and the extent to which the Agile best practices have been embraced. [..]The available questionnaire is presently basic, and is a long way off from providing the validation required to enrich

the characterisation of Use Case scenarios (as a contribution towards Phase 2 trials) and generally boost GE uptake.

Observation by FI-WARE

-          The D.10.5.1 deliverables states: "The design phase occurs taking care also requirements that have been successfully communicated from the Use Cases Projects to the FI-WARE chapters." - which does not imply a given fact that requirements actually were successfully communicated and does not imply that these requirements can be tracked throughout the whole process in the current way, agile is implemented and "lived". The deliverable in this point may have wrongly lead to the assumption that there were successfully communicated requirements, which the validation team itself couldn't really judge or imply. As the deliverable outlined later there is no tight linkage between defined requirements and the features provided.

-          Secondly the decision taken by the Architecture board reached was bond to the fact that validation based on all relevant features/epics provided by the FI-WARE project was not reasonable for the given amount of time and expected efforts - As of today FI-WARE features do comprise of more than 900 features (please bear in mind that only limited resources were foreseen in the DoW to actually execute on the validation task towards the core platform).

-          Finally it was decided by the Architecture board that a "validation questionnaire" to be provided by FI-WARE has to cater for validation and mainly will be based upon questions related to "validation context" and "generic enablers" and not based on features and requirements. The use case projects therefore were only bound to give the scenarios and their descriptions to FI-WARE and not their requirements.

-          Enrichment of the characterization of use case scenarios and boost of GE uptake was not in scope of the validation approach as implied in the reviewer statement.

-          As the amendment 4 foresees there will be again a reiteration of the validation process towards the phase 2 projects and their trials, where further changes in the process are currently being considered by the architecture board, potentially leading to a tighter linkage between requirements and GEs which might lead to a better validation. E.g. it is foreseen that use case projects will input their requirements to GE-based trackers and not to a common fi-ware tracker any more.



One of the feedbacks received by the use case projects almost speaks for itself:

As a general impression, the validation questions were quite helpful to provide feedback. The only issue we faced in our case was due to the fact that we evaluated the GEs per prototype component and not for specific scenarios.

4. Additionally, how testing and evaluation would be conducted in relation to the non-functional capabilities that are listed for the first releases in the Technical Roadmap is yet to be described.


Observation by FI-WARE

-          Checking non-functional capabilities was again reiterated within the architecture board and no common consensus could be found towards the validation process. This recommendation will be taken into the next discussion which will redefine the process for FIWARE v2 (but will not be part of the D.10.5.2 deliverable as for time constraints: phase 2 projects did not start to redefine the validation process and potentially won't do so for the next two month )




Here is again the complete text from the Review Report (only broken up with some newlines):


D10.5.1 Report on Validation Process including Validation with Use Case projects
This deliverable outlines the designed and recommended validation process for the use cases to follow. Additionally the initial feedback survey, which was initiate and send to the use
case projects and the main findings are outlined.
The validation process described in the document is generally well thought and detailed; however,
it has been devised without sufficient consideration of the FI-WARE project and FI-WARE Releases.

The validation approach is also considered insufficient,
in view of what is envisaged in the DoW in supporting Use Case projects on
deployment, execution and validation of the conceptual prototypes in respect of the available GEs.

According to the deliverable, the design phase of FI-WARE incorporates requirements that have been successfully communicated from the Use Cases Projects to the FI-WARE chapters. As the
link between Use Case requirements and
the actual content of the individual chapters is not readily traceable, this has a significant impact on the validation, and the extent to which the Agile best practices have been embraced.

As explained in the document, there is no tight linkage between the defined requirements and the features provided by the GE providers.
Hence, the validation and requirements evaluation will not be based on a requirements matrix, but will follow an open questionnaire approach.  The available questionnaire is presently basic,
and is a long way off from providing the validation required to enrich
the characterisation of Use Case scenarios (as a contribution towards Phase 2 trials) and generally boost GE uptake.

Additionally, how testing and evaluation would be conducted in relation to the non-functional capabilities that are listed for the first releases in the Technical Roadmap is yet to be described.



--

Thorsten Sandfuchs

SAP AG | Vincenz-Priessnitz-Strasse 1 | D-76131 Karlsruhe, Germany | www.sap.com<http://www.sap.com/>


Pflichtangaben/Mandatory Disclosure Statements: http://www.sap.com/company/legal/impressum.epx

Diese E-Mail kann Betriebs- oder Geschäftsgeheimnisse oder sonstige vertrauliche Informationen enthalten. Sollten Sie diese E-Mail irrtümlich erhalten haben, ist Ihnen eine Kenntnisnahme des Inhalts, eine Vervielfältigung oder Weitergabe der E-Mail ausdrücklich untersagt. Bitte benachrichtigen Sie uns und vernichten Sie die empfangene E-Mail. Vielen Dank.

This e-mail may contain trade secrets or privileged, undisclosed, or otherwise confidential information. If you have received this e-mail in error, you are hereby notified that any review, copying, or distribution of it is strictly prohibited. Please inform us immediately and destroy the original transmittal. Thank you for your cooperation.

Please consider the environment before printing this mail!


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/old-fiware-testbed/attachments/20130429/aa59022e/attachment.html>