Hi all, Regarding this issue, it must be said the new cluster for storage is finally working. This new cluster is based on a modern Hadoop and the problem regarding offsets in WebHDFS should be fixed. As said, this cluster is only suited for storage, not computing (a second cluster specifically addressing computing will be released in a few weeks). So, if your use case is about storing data in HDFS and consuming it through WebHDFS, with no Hive, no MapReduce, then you are ready to move from the ³old² cluster (cosmos.lab.fiware.org) to the new one (storage.cosmos.lab.fiware.org). If you are interested, let me now and I¹ll provision the account and I¹ll migrate the data from one cluster to the other. Regards, Francisco El 16/12/15 8:56, "FRANCISCO ROMERO BUENO" <francisco.romerobueno at telefonica.com> escribió: >The deployment started several months ago, this is a very ambitious task. >We are in the latest steps, most probably it will be ready after the Xmas >holidays. > >Regards, >Francisco > >El 16/12/15 8:38, "Manuel Escriche (JIRA)" <jira-help-desk at fi-ware.org> >escribió: > >> >> [ >>https://jira.fiware.org/browse/HELP-5439?page=com.atlassian.jira.plugin.s >>y >>stem.issuetabpanels:comment-tabpanel&focusedCommentId=38443#comment-38443 >>] >> >>Manuel Escriche commented on HELP-5439: >>--------------------------------------- >> >>Dear Andrea, >>Thanks for feedback! >> >>Dear Francisco, >>When do you foresee to deploy again? Release5.1? >>Thanks in advance! >> >>Kind regards, >>Manuel >> >> >>> [Fiware-tech-help] Issue on Cosmos Global Instance: cannot transfer >>>file parts >>> >>>------------------------------------------------------------------------ >>>- >>>----- >>> >>> Key: HELP-5439 >>> URL: https://jira.fiware.org/browse/HELP-5439 >>> Project: Help-Desk >>> Issue Type: extRequest >>> Components: FIWARE-TECH-HELP >>> Reporter: FW External User >>> Assignee: Francisco Romero >>> >>> Dear FIWARE Staff, >>> I would like to report a potential issue on the *Cosmos Global >>>Instance*. >>> We are using Cosmos to store input files for Hadoop jobs and their >>>related >>> output. We are transferring them from/to a remote back-end through the >>> WebHDFS API. >>> During outbound transfers of large files from Cosmos, the need to split >>>the >>> file content in parts has risen. If we use a simple OPEN operation like >>> *http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN * >>> the transfer is interrupted after 2 minutes even if the response has >>>not >>> transferred completely to the client. This should due to the timeout >>> configuration of the HTTP Server deployed on Cosmos. >>> As suggested by the WebHDFS API Doc >>> <https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#OPEN>, one should >>> consider to transfer single parts of the requested file by setting >>>*offset* >>> and *length* parameters to complete the request successfully. >>> The issue we found is that the *length* parameter seems to be ignored >>>by >>> the HTTP Server. >>> For example, if we execute the following >>> *http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&offset=100&length=10* >>> on a 200 bytes file, we would get the last 100 bytes of the file >>>itself, >>> and not bytes from 101 to 110. >>> We checked the current version of Hadoop deployed on Cosmos and we >>>found >>> out that release *0.20.2-cdh3u6 *is currently running. By giving a look >>>to >>> the HDFS changelogs >>> >>><https://hadoop.apache.org/docs/r0.23.11/hadoop-project-dist/hadoop-hdfs >>>/ >>>CHANGES.txt>, >>> there is an improvement (HDFS-3794) committed in release 0.23.3 that >>>could >>> be potentially related to the reported occurrence. >>> Best Regards, >>> Andrea Sassi >>> Since January 1st, old domains won't be supported and messages sent to >>>any domain different to @lists.fiware.org will be lost. >>> Please, send your messages using the new domain >>>(Fiware-tech-help at lists.fiware.org) instead of the old one. >>> _______________________________________________ >>> Fiware-tech-help mailing list >>> Fiware-tech-help at lists.fiware.org >>> https://lists.fiware.org/listinfo/fiware-tech-help >>> [Created via e-mail received from: Andrea Sassi >>><brainswitch at gmail.com>] >> >> >> >>-- >>This message was sent by Atlassian JIRA >>(v6.4.1#64016) > ________________________________ Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it. Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy