[Fiware-tech-help] Continues Upstream problems

[michael.ulbig at ptb.de]

Dear FIWARE support,

Our FIWARE-Setup:
- FIWARE-LD in Docker
- on an VM
  - memory 4GB
  - Disc: 50GB SSD
{
  "orionld version": "post-v1.5.1",
  "orion version": "1.15.0-next",
  "uptime": "0 d, 1 h, 4 m, 49 s",
  "git_hash": "nogitversion",
  "compile_time": "Sun Jan 28 08:20:37 UTC 2024",
  "compiled_by": "root",
  "compiled_in": "",
  "release_date": "Sun Jan 28 08:20:37 UTC 2024",
  "doc": "https://fiware-orion.readthedocs.org/en/master/"
}

Our UseCase:
We want to collect and distribute data from out electricity meters over 
FIWARE. 
For this we have a python script on a VM which collects the data, 
transforms it into ThreePhaseMeasurement Json-LD file and then sends it to 
the FIWARE on a different VM.
With this set up, we transfer so far about 30 Measurements to FIWARE every 
15 seconds.
In our attempts we tried to find a solution as simple as possible.

In our first attempt we wanted to save each measurement in each its own 
json-ld file.
Each file then has its own id (ndgsi-ld/v1/devicename-timestamp). 
The idea behind this, was to be able create with Json-ld and the MongoDB a 
semantic web / ontology. 
The problem we ran into with this is a linear increase usage of the memory 
(ram) until our fiware service breaks down.
After tring different things we came to the realization that we using 
FIWARE wrong and started our second attempt, 

In our second attempt we just work with one json-ld for each measurement 
and then patch the latest data.
Unfortunately, we still run in the same problem.
We also have tried to manually close the connections (Header {'Connection' 
:' Close'}, / Python response.close()).
Unfortunately with no effect.
Our memory still seems to continue soaking up data and we are still 
running in an out of memory:

Our question:
How are we meant to transfer data to FIWARE?
Are we on the right way by using Patch to upload data to FIWARE?
Do you maybe have an idea what causes this linear increase of memory 
usage?

We are thankful for any help or hint.
If you need any more information let me know 

With kind regards and thanks in advance
Michael Ulbig


Michael Ulbig
Physikalisch-Technische Bundesanstalt
Fachbereich 9.4 Metrologie für die digitale Transformation
Bundesallee 100
38116 Braunschweig
Telefon: +49 531 592 9407 
E-Mail: michael.ulbig at ptb.de 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-tech-help/attachments/20240619/df86218c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 18014 bytes
Desc: not available
URL: <https://lists.fiware.org/private/fiware-tech-help/attachments/20240619/df86218c/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 2092 bytes
Desc: not available
URL: <https://lists.fiware.org/private/fiware-tech-help/attachments/20240619/df86218c/attachment-0003.gif>