[Fiware-lab-help] Technical questions about application development with GEs

[FRANCISCO ROMERO BUENO]

Hi Florian,

Regarding your second question, one of the data analyzing GEs is the Big Data one. In a few words, the purpose of this GE is to store large volumes of data and expose mechanisms for querying and processing them. This is a GE consuming a lot of resources (a cluster of machines supports everything in the background) and thus it cannot be instantiated right now, but used in a shared way through the global instance we have deployed in FILAB. You can access it by requesting me a user account.

Once you have registered, if you want to use it you'll have to take into account the data lifecycle within this GE:

1. The data must be injected in the cluster. This means the data is distributed all along the cluster, not copied in a single machine. This can be done in two ways:
1.1. Static uploading of data files. A SFTP server is available in order to put your data in the cluster.
1.2. Dynamic uploading of data events. Using the Contex Broker GE you can register context information and send real time updates for this context info that are automatically recorded in the Big Data GE.
2. The data can be processed by performing MapReduce-based jobs, i.e. special Java-based applications based on applying a common operation over all the data -Map step- and the joining all the partial results -Reduce step-. This is suited if wanting to filter, transform, translate, aggregate, join, whatever you can imagine in a MapReduce schema. Nevertheless, this point is optional due to the characteristics of the querying mechanism of point 3.
3. The data can be queried using SQL-like interfaces, mainly, Hive queries. Out Big Data GE is based on the Hadoop ecosystem, and thus, Hive is available as a service that can be remotely accessed by a client of your own. When performing Hive queries, pre-defined MapReduce jobs are executed transparently to the user. For instance, if selecting a specific field of the data, a filter-like MapReduce is automatically executed on behalf of the user.

You can learn more about this GE in these selected links:

* How to upload data and program your own MapReduce jobs:
https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/BigData_Analysis_-_User_and_Programmer_Guide
* How to create Hive-based clients (this is from another Hackathon we organized some weeks ago):
http://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/Cosmos_in_the_FI-WARE_Smart_Cities_event_in_Santander
* More information on the available APIs/interfaces:
http://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/BigData_Analysis_Open_RESTful_API_Specification_%28PRELIMINARY%29
* Big Data GE general information:
https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.Data.BigData

Any question you may have regarding the GE internals, the design of your application with respect to the GE, your data, etc. please contact with me.

Regards,
Francisco


-----Mensaje original-----
De: fiware-lab-help-bounces at lists.fi-ware.eu [mailto:fiware-lab-help-bounces at lists.fi-ware.eu] En nombre de Florian Klemt
Enviado el: viernes, 15 de noviembre de 2013 18:45
Para: fiware-lab-help at lists.fi-ware.eu
Asunto: [Fiware-lab-help] Technical questions about application development with GEs

Hi there,

I would like to enter the Campus Labs Fi-Ware Challenge and have some technical questions about GEs.
The application that I would like to develop is a web based application (desktop and mobile) that is also capable of analysing user activities and recommend actions according to the previous user activities.

The first question is about running the necessary server infrastructure on GEs. As far as I understand, I can start machine instances that run in the Fi-Ware cloud. How do I provision these virtual machines to host the server software (web server, databases, etc.) as well as my application source code? Is it recommended to use tools like chef or puppet?

The second question is about analysing and processing user data and user actions. I read in your wiki that I can run GEs with that capability but didn't understand how I can include that into my application. Some further documentation or example code would be very helpful.

Thank you very much and best regards
- Florian Klemt
_______________________________________________
Fiware-lab-help mailing list
Fiware-lab-help at lists.fi-ware.eu
https://lists.fi-ware.eu/listinfo/fiware-lab-help

________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx