Dear FIWARE Staff, I would like to report a potential issue on the *Cosmos Global Instance*. We are using Cosmos to store input files for Hadoop jobs and their related output. We are transferring them from/to a remote back-end through the WebHDFS API. During outbound transfers of large files from Cosmos, the need to split the file content in parts has risen. If we use a simple OPEN operation like *http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN * the transfer is interrupted after 2 minutes even if the response has not transferred completely to the client. This should due to the timeout configuration of the HTTP Server deployed on Cosmos. As suggested by the WebHDFS API Doc <https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#OPEN>, one should consider to transfer single parts of the requested file by setting *offset* and *length* parameters to complete the request successfully. The issue we found is that the *length* parameter seems to be ignored by the HTTP Server. For example, if we execute the following *http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&offset=100&length=10* on a 200 bytes file, we would get the last 100 bytes of the file itself, and not bytes from 101 to 110. We checked the current version of Hadoop deployed on Cosmos and we found out that release *0.20.2-cdh3u6 *is currently running. By giving a look to the HDFS changelogs <https://hadoop.apache.org/docs/r0.23.11/hadoop-project-dist/hadoop-hdfs/CHANGES.txt>, there is an improvement (HDFS-3794) committed in release 0.23.3 that could be potentially related to the reported occurrence. Best Regards, Andrea Sassi -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.fiware.org/private/fiware-tech-help/attachments/20151201/38e5c157/attachment.html>
You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy Cookies policy