[英]How to implement an HTTP transfer of huge binary files from the file generator to the server (Java)?
Simply put our system consists of a Server and an Agent. 简单地说,我们的系统包括服务器和代理。 The Agent generates a huge binary file, which may be required to be transfered to the Server.
代理生成一个巨大的二进制文件,可能需要将其转移到服务器。
Given: 鉴于:
I am looking for techniques and libraries to help me with transfering huge files. 我正在寻找技术和库来帮助我传输大文件。 Some of the topics, which I am aware of are:
我所知道的一些主题是:
Has anyone encountered this problem in their products and how was it dealt with? 有人在他们的产品中遇到过这个问题吗?它是如何处理的?
Some may question the choice of HTTP as the transfer protocol. 有些人可能会质疑HTTP作为传输协议的选择。 The thing is that the Server and the Agent may be quite remoted from each other, even if located in the same corporate network.
问题是,即使位于同一公司网络中,服务器和代理也可能相互远程连接。 We have already faced numerous issues related to the fact that customers keep only HTTP ports open on the nodes in their corporate networks.
我们已经面临许多与客户仅在其公司网络中的节点上打开HTTP端口这一事实相关的问题。 It does not leave us much choice, but use HTTP.
它不会给我们太多选择,但使用HTTP。 Using FTP is fine, but it will have to be tunneled through HTTP - does it mean we still have all the benefits of FTP or will it cripple it to the point where other alternatives are more viable?
使用FTP很好,但它必须通过HTTP进行隧道传输 - 它是否意味着我们仍然拥有FTP的所有好处,还是会将其削弱到其他替代方案更可行的程度? I do not know.
我不知道。
Correction - HTTPS is always open and sometimes (but not always) HTTP is open as well. 更正 - HTTPS始终是打开的,有时(但不总是)HTTP也是打开的。 But that is it.
但就是这样。
You can use any protocol on port 80. Using HTTP is a good choice, but you don't have to use it. 您可以在端口80上使用任何协议。使用HTTP是一个不错的选择,但您不必使用它。
Compression Which one to choose?
压缩哪一个选择? We do not limit ourselves to gzip or deflate, just because they are the most popular for HTTP traffic.
我们不限于gzip或deflate,只是因为它们是最流行的HTTP流量。 If there is some unusual compression scheme, which yields better results for our task - so be it.
如果有一些不寻常的压缩方案,这对我们的任务产生更好的结果 - 所以就这样吧。
The best compression depends on the content. 最佳压缩取决于内容。 I would use Deflator for simplicity, however BZIP2 can give better results (requires a library)
我会使用Deflator来简化,但是BZIP2可以提供更好的结果(需要一个库)
For your file type you may find doing some compression specific to that type first, can make the data sent smaller. 对于您的文件类型,您可能会发现首先对该类型进行一些特定的压缩,可以使数据发送得更小。
Splitting Obviously, the file needs to be split and transfered in several parallel sessions.
拆分显然,文件需要在几个并行会话中拆分和转移。
This is no obvious to me. 这对我来说并不明显。 Downloading data in parallel improves performance by grabbing more of the available bandwidth (ie squeezing out other users of the same bandwidth) This may be undesirable or even pointless (if there are no other users)
并行下载数据可以通过获取更多可用带宽来提高性能(即挤出相同带宽的其他用户)这可能是不受欢迎的甚至是无意义的(如果没有其他用户)
Background Transfering a huge file takes a long time.
背景传输大文件需要很长时间。 Does it affect the solution, if at all?
它是否会影响解决方案,如果有的话?
You will want the ability to re-start the download at any point. 您将希望能够随时重新开始下载。
Security Is HTTPS the way to go?
安全性HTTPS是否可行? Or should we take another approach, given the volume of data?
或者,考虑到数据量,我们应该采取另一种方法吗?
I am sure its fine, regardless of the volume of data. 不管数据量多少,我都确定没问题。
off-the-shelf I am fully prepared to code it myself (should be fun), but I cannot avoid the question whether there are any off-the-shelf solutions satisfying my demands.
现成的我已经准备好自己编写代码(应该很有趣),但我无法回避是否有任何现成的解决方案满足我的要求。
I would try using existing web servers to see if they are up to the job. 我会尝试使用现有的Web服务器来查看他们是否能胜任这项工作。 I would be surprised if there isn't a free web server which does all the above.
如果没有一个免费的网络服务器可以完成上述所有工作,我会感到惊讶。
Here is a selection http://www.java-sources.net/open-source/web-servers 这是一个选择http://www.java-sources.net/open-source/web-servers
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.