[Fiware-tech-help] [FI-WARE-JIRA] (HELP-5439) Issue on Cosmos Global Instance: cannot transfer file parts

[Andrea Sassi]

Thanks Francisco,

If you do not mind, I would like to discuss another point.

We decided to develop a Spark pipeline of tasks to process our data.
Currently we are using our private Hadoop single node instance to execute
the Spark jobs. Unfortunately, only recently we found out that Spark is not
deployed on Cosmos, but we need a full integration of the Cosmos GE within
our architecture. I'm not sure that making our product viable by using a
plain private Hadoop will be enough.

I found that Apache Oozie can schedule
<https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html> Spark
jobs and I was wondering if in the next Hadoop deployment on Cosmos also
Spark will be installed.

Thanks in advance for your collaboration.

Regards,
Andrea

2015-12-02 11:39 GMT+01:00 FRANCISCO ROMERO BUENO <
francisco.romerobueno at telefonica.com>:

> Dear Andrea,
>
> As you have noticed, this is a problem regarding the Hadoop version
> (pretty old) we are running at the global instance. This will be fixed
> once we release the new global instance (end of the year/start of the new
> one), running a modern version of Hadoop based on Hortonworks.
>
> Thanks for the feedback.
>
> Regards,
> Francisco
>
> El 1/12/15 17:05, "Manuel Escriche (JIRA)" <jira-help-desk at fi-ware.org>
> escribió:
>
> >
> >     [
> >
> https://jira.fiware.org/browse/HELP-5439?page=com.atlassian.jira.plugin.sy
> >stem.issuetabpanels:all-tabpanel ]
> >
> >Manuel Escriche reassigned HELP-5439:
> >-------------------------------------
> >
> >    Assignee: Francisco Romero
> >
> >> [Fiware-tech-help] Issue on Cosmos Global Instance: cannot transfer
> >>file parts
> >>
> >>-------------------------------------------------------------------------
> >>-----
> >>
> >>                 Key: HELP-5439
> >>                 URL: https://jira.fiware.org/browse/HELP-5439
> >>             Project: Help-Desk
> >>          Issue Type: extRequest
> >>          Components: FIWARE-TECH-HELP
> >>            Reporter: FW External User
> >>            Assignee: Francisco Romero
> >>
> >> Dear FIWARE Staff,
> >> I would like to report a potential issue on the *Cosmos Global
> >>Instance*.
> >> We are using Cosmos to store input files for Hadoop jobs and their
> >>related
> >> output. We are transferring them from/to a remote back-end through the
> >> WebHDFS API.
> >> During outbound transfers of large files from Cosmos, the need to split
> >>the
> >> file content in parts has risen. If we use a simple OPEN operation like
> >> *http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN *
> >> the transfer is interrupted after 2 minutes even if the response has not
> >> transferred completely to the client. This should due to the timeout
> >> configuration of the HTTP Server deployed on Cosmos.
> >> As suggested by the WebHDFS API Doc
> >> <https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#OPEN>, one should
> >> consider to transfer single parts of the requested file by setting
> >>*offset*
> >> and *length* parameters to complete the request successfully.
> >> The issue we found is that the *length* parameter seems to be ignored by
> >> the HTTP Server.
> >> For example, if we execute the following
> >> *http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&offset=100&length=10*
> >> on a 200 bytes file, we would get the last 100 bytes of the file itself,
> >> and not bytes from 101 to 110.
> >> We checked the current version of Hadoop deployed on Cosmos and we found
> >> out that release *0.20.2-cdh3u6 *is currently running. By giving a look
> >>to
> >> the HDFS changelogs
> >>
> >><
> https://hadoop.apache.org/docs/r0.23.11/hadoop-project-dist/hadoop-hdfs/
> >>CHANGES.txt>,
> >> there is an improvement (HDFS-3794) committed in release 0.23.3 that
> >>could
> >> be potentially related to the reported occurrence.
> >> Best Regards,
> >> Andrea Sassi
> >> Since January 1st, old domains won't be supported and messages sent to
> >>any domain different to @lists.fiware.org will be lost.
> >> Please, send your messages using the new domain
> >>(Fiware-tech-help at lists.fiware.org) instead of the old one.
> >> _______________________________________________
> >> Fiware-tech-help mailing list
> >> Fiware-tech-help at lists.fiware.org
> >> https://lists.fiware.org/listinfo/fiware-tech-help
> >> [Created via e-mail received from: Andrea Sassi <brainswitch at gmail.com
> >]
> >
> >
> >
> >--
> >This message was sent by Atlassian JIRA
> >(v6.4.1#64016)
>
>
> ________________________________
>
> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
> puede contener información privilegiada o confidencial y es para uso
> exclusivo de la persona o entidad de destino. Si no es usted. el
> destinatario indicado, queda notificado de que la lectura, utilización,
> divulgación y/o copia sin autorización puede estar prohibida en virtud de
> la legislación vigente. Si ha recibido este mensaje por error, le rogamos
> que nos lo comunique inmediatamente por esta misma vía y proceda a su
> destrucción.
>
> The information contained in this transmission is privileged and
> confidential information intended only for the use of the individual or
> entity named above. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this transmission in error, do not read it. Please immediately reply to the
> sender that you have received this communication in error and then delete
> it.
>
> Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário,
> pode conter informação privilegiada ou confidencial e é para uso exclusivo
> da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário
> indicado, fica notificado de que a leitura, utilização, divulgação e/ou
> cópia sem autorização pode estar proibida em virtude da legislação vigente.
> Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique
> imediatamente por esta mesma via e proceda a sua destruição
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-tech-help/attachments/20151202/70160363/attachment.html>