optimizing file writes with zlib?

Question

I profiled my app and apparently 37% of the time in my app is in ostream::write. Less then 7% was actually compressing it (deflate_slow is the function name it gave me).

How can i optimize file IO with zlib? should i write my own fopen/fwrite/fread/fclose wrapper? Is there a wrapper that exist? can i do something with the buffer size (i would like it large). What can i do to make writing faster?

Answer 1

First, check that ostream::write is called inefficiently. Depending on your platform, you should use:

• Windows: procmon.exe (from the SysInternals Suite)
• Linux: strace
• Solaris: ktruss
• NetBSD: ktrace

Check whether there are many calls to WriteFile (Windows) or write (POSIX-like) which only write a few bytes, namely less than 100. If you cannot find these, it might not help writing a wrapper.

The above tools can also measure whether you lose time in the operating system's kernel or in the user code. That should give you a hint as to where you can improve performance.