[Fiware-tech-help] CEF funded project compliance tests

[Elena Fulvio]

Good evening,

we are trying to run the conformance test for the Orion Context Broker that is included in a project funded by CEF, but we are facing some difficulties.


We have tried to reach you on last friday but the email was "bounced" because of the attachments size (they were the complete logs from the tests suites ):


Your mail to 'Fiware-tech-help' with the subject

    CEF funded project compliance tests

Is being held until the list moderator can review it for approval.

The reason it is being held:

    Message body is too big: 702129 bytes with a limit of 500 KB

Either the message will get posted to the list, or you will receive
notification of the moderator's decision.  If you would like to cancel
this posting, please visit the following URL:


We are running the Orion Context Broker via Kubernetes. The context broker was installed by helm (following the instruction at https://github.com/FIWARE/helm-charts ), resulting in a container based on the image quay.io/fiware/orion-ld:1.1.0 (for reference the mongodb image, installed along via the helm chart is quay.io/opencloudio/ibm-mongodb:4.0.24). The containers are up and running and we are able to successfully execute the REST apis described in https://fiware-orion.readthedocs.io/en/master/ , however the test suites we are running seem to fail.


We found two different test suites, the first one (NGSI-LD Test Suite) which is linked in CEF digital page https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/Conformance+testing and the second one (ETSI NGSI-LD API Conformance Test Suite) on the FIWARE github page.
[https://ec.europa.eu/cefdigital/wiki/cefdigital/wiki/rest/documentConversion/latest/conversion/thumbnail/356876566/1]<https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/Conformance+testing>
Conformance testing - ec.europa.eu<https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/Conformance+testing>
CEF Context Broker offers two different types of testing services:
ec.europa.eu


In spite of the fact that the first one seems to be deprecated and the second one is still under development we tried both of them:

[cid:9e8bbe15-ba2d-4e5a-8bb6-5c5595028f9c]
      
      NGSI-LD Test Suite:

            We downloaded the test suite from : https://github.com/FIWARE/NGSI-LD_TestSuite


            The first test was directed to the helm instance I mentioned before and resulted in :

            Test Suites: 15 failed, 22 passed, 37 total
            Tests:       49 failed, 120 passed, 169 total
            Snapshots:   0 total
            Time:        117.426s, estimated 118s
      
            which is odd, considering the fact that the Orion Context Broker instance we are testing is the default image provided by the official helm chart (the complete logs are available in the attachment OUTPUT_NGIS-LD_helm.txt).

            We also tried to run the same test suite using the docker containers that are provided in the suite itself (under /docker-compose/orion.yml) and it resulted in:

            Test Suites: 11 failed, 26 passed, 37 total
            Tests:       39 failed, 130 passed, 169 total
            Snapshots:   0 total
            Time:        102.645s

            which is extremely odd since in this case Orion-LD, mongodb, accumulator server & notification server are all provided by the suite itself .

      ETSI Test Suite:

            We downloaded the test suite from:  https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/tree/develop  since the master branch is still empty, and we tried to test the helm instance again.
            We were actually able to launch part of the tests but the testing process seems to freeze at some point during the execution.




Considering all of the above, how we can achieve the CEF compliance for the FIWARE Orion Context broker implementation? Are those tests sufficient to be compliant?


Thank you for your time and sorry for the long email.


Best regards,



Elena Fulvio
Solution Architect

KEY TO BUSINESS S.R.L.

[https://www.key2.it/img/loghi_mail/Email.png]  e.fulvio at key2.it<mailto:e.fulvio at key2.it>
[https://www.key2.it/img/loghi_mail/Sito%20web.png]     https://www.key2.it/<https://www.key2.it/it/>
[https://www.key2.it/img/loghi_mail/Indirizzo.png]      Viale dell'Esperanto, 71, 00144 Roma (RM)
Via Lunigiana 40, 20125 Milano (MI)
SS 17 BIS SNC, 67100 L'Aquila (AQ)
[https://www.key2.it/img/loghi_mail/logo_k2_blue_xxs.png]       [https://www.key2.it/img/loghi_mail/Instagram.png] <https://www.instagram.com/k2business/?hl=it>  [https://www.key2.it/img/loghi_mail/Facebook.png] <https://www.facebook.com/key2.it>  [https://www.key2.it/img/loghi_mail/Linkedin.png] <https://www.linkedin.com/company/key-2/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-tech-help/attachments/20221110/76bace96/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 157871 bytes
Desc: image.png
URL: <https://lists.fiware.org/private/fiware-tech-help/attachments/20221110/76bace96/attachment-0001.png>