[英]Force flush on a GZIPOutputStream in java
we are working on a program where we need to flush (force compress and send data) a GZIPOutputStream.我们正在开发一个需要刷新(强制压缩和发送数据)GZIPOutputStream 的程序。 The problem is, that the flush method of the GZIPOutputStream doesn't work as expected (force compress and send data), instead the Stream waits for more data for efficient data compression.
问题是, GZIPOutputStream 的刷新方法没有按预期工作(强制压缩并发送数据),而是 Stream 等待更多数据以进行有效的数据压缩。
When you call finish the data is compressed and sent over the output stream but the GZIPOutputStream (not the underlying stream) will be closed so we cant write more data till we create a new GZIPOutputStream, which costs time and performance.当您调用完成时,数据被压缩并通过输出流发送,但 GZIPOutputStream(不是底层流)将关闭,因此我们无法写入更多数据,直到我们创建新的 GZIPOutputStream,这会消耗时间和性能。
Hope anyone can help with this.希望任何人都可以帮助解决这个问题。
Best regards.最好的问候。
I haven't tried this yet, and this advice won't be useful until we have Java 7 in hand, but the documentation for GZIPOutputStream
's flush()
method inherited from DeflaterOutputStream
relies upon the flush mode specified at construction time with the syncFlush
argument (related to Deflater#SYNC_FLUSH
) to decide whether to flush the pending data to be compressed.我还没有尝试过这个,这个建议在我们手头有 Java 7 之前不会有用,但是从
DeflaterOutputStream
继承的GZIPOutputStream
的flush()
方法的文档依赖于在构建时使用syncFlush
指定的刷新模式参数(与Deflater#SYNC_FLUSH
相关)来决定是否刷新待压缩的待处理数据。 This syncFlush
argument is also accepted by GZIPOutputStream
at construction time.这个
syncFlush
参数在构建时也被GZIPOutputStream
接受。
It sounds like you want to use either Deflator#SYNC_FLUSH
or maybe even Deflater#FULL_FLUSH
, but, before digging down that far, first try working with the two-argument or the four-argument GZIPOutputStream
constructor and pass true
for the syncFlush
argument.听起来您想使用
Deflator#SYNC_FLUSH
甚至Deflater#FULL_FLUSH
,但是,在深入研究之前,首先尝试使用两个参数或四个参数的GZIPOutputStream
构造函数,并为syncFlush
参数传递true
。 That will activate the flushing behavior you desire.这将激活您想要的冲洗行为。
I didn't find the other answer to work.我没有找到其他工作答案。 It still refused to flush because the native code that GZIPOutputStream is using holds onto the data.
它仍然拒绝刷新,因为 GZIPOutputStream 使用的本机代码保留了数据。
Thankfully, I discovered that someone has implemented a FlushableGZIPOutputStream as part of the Apache Tomcat project.幸运的是,我发现有人实现了 FlushableGZIPOutputStream 作为 Apache Tomcat 项目的一部分。 Here is the magic part:
这是神奇的部分:
@Override
public synchronized void flush() throws IOException {
if (hasLastByte) {
// - do not allow the gzip header to be flushed on its own
// - do not do anything if there is no data to send
// trick the deflater to flush
/**
* Now this is tricky: We force the Deflater to flush its data by
* switching compression level. As yet, a perplexingly simple workaround
* for
* http://developer.java.sun.com/developer/bugParade/bugs/4255743.html
*/
if (!def.finished()) {
def.setLevel(Deflater.NO_COMPRESSION);
flushLastByte();
flagReenableCompression = true;
}
}
out.flush();
}
You can find the entire class in this jar (if you use Maven):你可以在这个 jar 中找到整个类(如果你使用 Maven):
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-coyote</artifactId>
<version>7.0.8</version>
</dependency>
Or just go and grab the source code FlushableGZIPOutputStream.java或者只是去获取源代码FlushableGZIPOutputStream.java
It's released under the Apache-2.0 license.它是在 Apache-2.0 许可下发布的。
This code is working great for me in my application.这段代码在我的应用程序中对我很有用。
public class StreamingGZIPOutputStream extends GZIPOutputStream {
public StreamingGZIPOutputStream(OutputStream out) throws IOException {
super(out);
}
@Override
protected void deflate() throws IOException {
// SYNC_FLUSH is the key here, because it causes writing to the output
// stream in a streaming manner instead of waiting until the entire
// contents of the response are known. for a large 1 MB json example
// this took the size from around 48k to around 50k, so the benefits
// of sending data to the client sooner seem to far outweigh the
// added data sent due to less efficient compression
int len = def.deflate(buf, 0, buf.length, Deflater.SYNC_FLUSH);
if (len > 0) {
out.write(buf, 0, len);
}
}
}
There is same problem on Android
also. Android
也有同样的问题。 Accepter answer doesn't work because def.setLevel(Deflater.NO_COMPRESSION);
接受者的回答不起作用,因为
def.setLevel(Deflater.NO_COMPRESSION);
throws exception.抛出异常。 According
flush
method it changes compress level of Deflater
.根据
flush
方法,它改变了Deflater
压缩级别。 So I suppose changing compression should be called before writing data, but I'm not sure.所以我认为应该在写入数据之前调用更改压缩,但我不确定。
There're 2 other options:还有 2 个其他选项:
Bug ID 4813885 handles this issue. 错误 ID 4813885处理此问题。 The comment of "DamonHD", submitted on 9 Sep 2006 (about halfway the bugreport) contains an example of
FlushableGZIPOutputStream
which he built on top of Jazzlib's net.sf.jazzlib.DeflaterOutputStream
. 2006 年 9 月 9 日提交的“DamonHD”评论(大约是错误报告的一半)包含一个
FlushableGZIPOutputStream
示例,该示例构建在Jazzlib 的net.sf.jazzlib.DeflaterOutputStream
之上。
For reference, here's a (reformatted) extract:作为参考,这里有一个(重新格式化的)摘录:
/**
* Substitute for GZIPOutputStream that maximises compression and has a usable
* flush(). This is also more careful about its output writes for efficiency,
* and indeed buffers them to minimise the number of write()s downstream which
* is especially useful where each write() has a cost such as an OS call, a disc
* write, or a network packet.
*/
public class FlushableGZIPOutputStream extends net.sf.jazzlib.DeflaterOutputStream {
private final CRC32 crc = new CRC32();
private final static int GZIP_MAGIC = 0x8b1f;
private final OutputStream os;
/** Set when input has arrived and not yet been compressed and flushed downstream. */
private boolean somethingWritten;
public FlushableGZIPOutputStream(final OutputStream os) throws IOException {
this(os, 8192);
}
public FlushableGZIPOutputStream(final OutputStream os, final int bufsize) throws IOException {
super(new FilterOutputStream(new BufferedOutputStream(os, bufsize)) {
/** Suppress inappropriate/inefficient flush()es by DeflaterOutputStream. */
@Override
public void flush() {
}
}, new net.sf.jazzlib.Deflater(net.sf.jazzlib.Deflater.BEST_COMPRESSION, true));
this.os = os;
writeHeader();
crc.reset();
}
public synchronized void write(byte[] buf, int off, int len) throws IOException {
somethingWritten = true;
super.write(buf, off, len);
crc.update(buf, off, len);
}
/**
* Flush any accumulated input downstream in compressed form. We overcome
* some bugs/misfeatures here so that:
* <ul>
* <li>We won't allow the GZIP header to be flushed on its own without real compressed
* data in the same write downstream.
* <li>We ensure that any accumulated uncompressed data really is forced through the
* compressor.
* <li>We prevent spurious empty compressed blocks being produced from successive
* flush()es with no intervening new data.
* </ul>
*/
@Override
public synchronized void flush() throws IOException {
if (!somethingWritten) { return; }
// We call this to get def.flush() called,
// but suppress the (usually premature) out.flush() called internally.
super.flush();
// Since super.flush() seems to fail to reliably force output,
// possibly due to over-cautious def.needsInput() guard following def.flush(),
// we try to force the issue here by bypassing the guard.
int len;
while((len = def.deflate(buf, 0, buf.length)) > 0) {
out.write(buf, 0, len);
}
// Really flush the stream below us...
os.flush();
// Further flush()es ignored until more input data data written.
somethingWritten = false;
}
public synchronized void close() throws IOException {
if (!def.finished()) {
def.finish();
do {
int len = def.deflate(buf, 0, buf.length);
if (len <= 0) {
break;
}
out.write(buf, 0, len);
} while (!def.finished());
}
// Write trailer
out.write(generateTrailer());
out.close();
}
// ...
}
You may find it useful.你可能会发现它很有用。
as @seh said, this works great:正如@seh 所说,这很好用:
ByteArrayOutputStream stream = new ByteArrayOutputStream();
// the second param need to be true
GZIPOutputStream gzip = new GZIPOutputStream(stream, true);
gzip.write( .. );
gzip.flush();
...
gzip.close()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.