如何调试 Apache 中损坏的响应标头？

Question

We're trying to get Webdav running in Kubernetes using an Azure Files storage backend, which is mounted in the container on /dav/data . The container itself is running Alpine Linux 3.12.1, in which we're installing all our required Apache packages.

All is well and good when not mounting this storage, or when using a different storage backend. However, when mounting the Azure Files storage, stuff starts to break.

Uploading files works without issue, but downloading does not; most software complains about invalid HTTP headers/responses. When investigating this further, I see that the beginning of the headers seems to be getting cut off.

Example headers of a correct response (obtained by not mounting the volume):

HTTP/1.1 200 OK
Date: Tue, 01 Dec 2020 13:54:53 GMT
Server: Apache/2.4.46 (Unix)
Last-Modified: Tue, 01 Dec 2020 13:51:02 GMT
ETag: "bla"
Accept-Ranges: bytes
Content-Length: 985
Connection: close

Example headers of an incorrect response:

s: bytes
Content-Length: 985
Connection: close

Everything up to the first s in the Accept-Ranges header seems to be getting eaten somewhere. There also seem to be a number of extra null bytes at the end of the response.

In an effort to get to the bottom of this I looked into logging as much as I possibly could, and stumbled upon the DumpIO module, which would allow me to log both the response headers as well as the body. For some reason, loading this module, setting DumpIOOutput On and LogLevel dumpio:trace7 actually fixes the issue. Response headers are fine, and the response body is exactly what you'd expect. And it's driving me nuts.

I suspect there's some kind of weird buffer/window issue being caused by an interaction between Apache and the mounted volume, but I haven't been able to figure out what.

We've since changed the storage backend used for the volume, but I'd still really like to know what caused this issue.

I've also been able to reproduce this locally in Docker.

Answer 1

Having the exact same issue after upgrading the AKS cluster from 1.17 to 1.18. Headers are malformed. Tried updating to a newer Apache version but doesn't work. Temporarily switched from azureFile to azureDisk and that works! Will see if I can create an AKS bug report for this

Answer 2

In case you haven't come across the culprit yet (or anyone else coming across this while googling like I did). The issue is Apache's EnableMMAP setting.

Solution: In your Apache conf, set EnableMMAP off .

Sources:

https://cloudiseasy.com/2021/06/13/deploying-apache-server-on-aks-with-azure-files/

https://httpd.apache.org/docs/2.4/mod/core.html#enablemmap

Apache is adding header to images resulting in corrupting images