PHP CURL导致巨大的Apache访问日志

Question

I'm curious to know how to stop Apache from logging every URL I search with CURL.

My PHP script opens a few hundred thousand URLs, scans them, takes a tiny bit of info, closes, and then opens the next.

I discovered after opening the access log that each and every URL opened with CURL is written to the access log.

::1 - - [01/Dec/2010:18:37:37 -0600] "GET /test.php HTTP/1.1" 200 8469 "-"..."

My access log is almost 45MBytes large. Help anyone?

Answer 1

This is the purpose for access log - recording any incoming traffic

In order to effectively manage a web server, it is necessary to get feedback about the activity and performance of the server as well as any problems that may be occurring. The Apache HTTP Server provides very comprehensive and flexible logging capabilities. This document describes how to configure its logging capabilities, and how to understand what the logs contain.

source: http://httpd.apache.org/docs/trunk/logs.html

Of course, you have the option to disable logging (preferable not)

Answer 2

See the Apache manual , about Conditional Logs . That may be what you are looking for.

Answer 3

If all of your curl requests are coming from a single or otherwise manageable group of IPs you can exclude them from your logs with a configuration similar to the following:

# Set your address here, you can do this for multiple addresses
SetEnvIf Remote_Addr "1\.1\.1\.1" mycurlrequest
CustomLog logs/access_log common env=!mycurlrequest

You can do something similar with the user agent field which by default will indicate that it's curl.

You can read more here: http://httpd.apache.org/docs/2.2/logs.html#accesslog (conditional logging is the last section under this header) and here http://httpd.apache.org/docs/2.2/mod/mod_setenvif.html#setenvif

If you want to conditionally exclude logging I would to it by the most precise method possible such as the ip address. In the event the server is externally accessible you probably don't want to find yourself NOT logging external requests from curl.

Using conditional logging you can also segment your logging if you want to multiple files one of which you could roll more frequently. The benefit of that is you can save space and at the same time have log data to help research and debug.