Program consumed memory keeps growing while using PyCurl in multi threaded environment

Question

Python program consumed memory grows over time when PyCurl is used in multi-threaded environment for concurrent HTTP API calls.

Attached Python code snippet demonstrates the problem. Program simply hits a URL in multiple threads using PyCurl and logs memory usage after every iteration.

Code

'''
Run pycurl in multi-threaded environment and log memory usage.
'''

import os
import io
import sys
import time
import logging
from multiprocessing.pool import ThreadPool

import pycurl
import psutil


# URL for testing
URL = 'https://github.com/pycurl/pycurl/blob/master/README.rst'

# Setup logger
LOGGER = logging.getLogger('TestPyCURL')
CONSOLE_HANDLER = logging.StreamHandler(sys.stdout)
LOG_FORMAT = '%(asctime)s - %(threadName)s - %(message)s'
FORMATTER = logging.Formatter(LOG_FORMAT, '%b %d %I:%M:%S')
CONSOLE_HANDLER.setFormatter(FORMATTER)
CONSOLE_HANDLER.setLevel(logging.DEBUG)
LOGGER.addHandler(CONSOLE_HANDLER)
LOGGER.setLevel(logging.DEBUG)


# Note current memory usage of program
process = psutil.Process(os.getpid())
last_usage = process.memory_info().rss
def log_memory_usage():
    '''
    Log change in memory usage of process.
    '''
    global last_usage
    usage = process.memory_info().rss
    change = usage - last_usage
    change_prcnt = change / last_usage * 100
    if usage == last_usage:
        slope = 'No change'
    elif usage > last_usage:
        slope = 'Increased'
    else:
        slope = 'Decreased'
    last_usage = usage
    LOGGER.info('Memory usage: Total: %s MB Change: %s, %s KB, %.3f %%',
                usage / 1000000, slope, change / 1000, change_prcnt)


def curl_request(url):
    '''
    Do HTTP request using Curl.
    '''
    buffer = io.BytesIO()
    curl = pycurl.Curl()
    curl.setopt(curl.URL, url)
    curl.setopt(curl.TIMEOUT, 10)
    curl.setopt(curl.CONNECTTIMEOUT, 5)
    curl.setopt(curl.WRITEDATA, buffer)
    try:
        curl.perform()
    except pycurl.error as exce:
        LOGGER.error('Request failed with %s', exce)
    else:
        content = buffer.getvalue()
        status_code = curl.getinfo(curl.RESPONSE_CODE)
        LOGGER.info('Status Code: %s', status_code)
        LOGGER.info('Content length: %s', len(content))
    curl.close()
    buffer.close()


def main():
    '''
    Test pycurl.
    '''
    # Create pool of worker threads
    pool = ThreadPool(100)
    # Run forever
    while True:
        # Log change in memory usage of program
        log_memory_usage()
        # Sleep
        time.sleep(3)
        # Launch parallel HTTP requests
        pool.map(curl_request, [URL for _ in range(0, 100)])


if __name__ == '__main__':
    main()

Test Environment

• OS: Ubuntu 20.04
• Python: 3.8.10
• PyCurl and LibCurl: PycURL/7.44.1 libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3

Memory usage over time logged by test program

Dec 10 11:18:14 - MainThread - Memory usage: Total: 22.48704 MB Change: Increased, 1425.408 KB, 6.768 %
Dec 10 11:18:48 - MainThread - Memory usage: Total: 44.01152 MB Change: Increased, 21524.48 KB, 95.719 %
Dec 10 11:19:22 - MainThread - Memory usage: Total: 60.547072 MB Change: Increased, 16535.552 KB, 37.571 %
Dec 10 11:19:56 - MainThread - Memory usage: Total: 63.95904 MB Change: Increased, 3411.968 KB, 5.635 %
Dec 10 11:20:30 - MainThread - Memory usage: Total: 68.304896 MB Change: Increased, 4345.856 KB, 6.795 %
Dec 10 11:20:54 - MainThread - Memory usage: Total: 69.722112 MB Change: Increased, 1417.216 KB, 2.075 %
Dec 10 11:21:28 - MainThread - Memory usage: Total: 70.934528 MB Change: Increased, 1212.416 KB, 1.739 %
Dec 10 11:21:41 - MainThread - Memory usage: Total: 72.00768 MB Change: Increased, 1073.152 KB, 1.513 %
Dec 10 11:22:25 - MainThread - Memory usage: Total: 72.64256 MB Change: Increased, 634.88 KB, 0.882 %
Dec 10 11:22:59 - MainThread - Memory usage: Total: 74.4448 MB Change: Increased, 1802.24 KB, 2.481 %
Dec 10 11:23:22 - MainThread - Memory usage: Total: 78.372864 MB Change: Increased, 3928.064 KB, 5.276 %
Dec 10 11:24:05 - MainThread - Memory usage: Total: 78.700544 MB Change: Increased, 327.68 KB, 0.418 %
Dec 10 11:24:39 - MainThread - Memory usage: Total: 79.03232 MB Change: Increased, 331.776 KB, 0.422 %
Dec 10 11:25:22 - MainThread - Memory usage: Total: 79.183872 MB Change: Increased, 151.552 KB, 0.192 %
Dec 10 11:25:41 - MainThread - Memory usage: Total: 95.555584 MB Change: Increased, 16371.712 KB, 20.676 %
Dec 10 11:26:15 - MainThread - Memory usage: Total: 97.001472 MB Change: Increased, 1445.888 KB, 1.513 %
Dec 10 11:26:58 - MainThread - Memory usage: Total: 100.442112 MB Change: Increased, 3440.64 KB, 3.547 %
Dec 10 11:27:17 - MainThread - Memory usage: Total: 101.5808 MB Change: Increased, 1138.688 KB, 1.134 %
Dec 10 11:28:01 - MainThread - Memory usage: Total: 102.989824 MB Change: Increased, 1409.024 KB, 1.387 %
Dec 10 11:28:25 - MainThread - Memory usage: Total: 103.448576 MB Change: Increased, 458.752 KB, 0.445 %
Dec 10 11:29:09 - MainThread - Memory usage: Total: 103.612416 MB Change: Increased, 163.84 KB, 0.158 %
Dec 10 11:29:22 - MainThread - Memory usage: Total: 109.068288 MB Change: Increased, 5455.872 KB, 5.266 %
Dec 10 11:29:33 - MainThread - Memory usage: Total: 112.205824 MB Change: Increased, 3137.536 KB, 2.877 %
Dec 10 11:29:41 - MainThread - Memory usage: Total: 112.8448 MB Change: Increased, 638.976 KB, 0.569 %
Dec 10 11:30:14 - MainThread - Memory usage: Total: 113.074176 MB Change: Increased, 229.376 KB, 0.203 %
Dec 10 11:30:38 - MainThread - Memory usage: Total: 113.082368 MB Change: Increased, 8.192 KB, 0.007 %
Dec 10 11:31:12 - MainThread - Memory usage: Total: 116.117504 MB Change: Increased, 3035.136 KB, 2.684 %
Dec 10 11:31:25 - MainThread - Memory usage: Total: 117.53472 MB Change: Increased, 1417.216 KB, 1.221 %
Dec 10 11:31:36 - MainThread - Memory usage: Total: 122.12224 MB Change: Increased, 4587.52 KB, 3.903 %
Dec 10 11:32:00 - MainThread - Memory usage: Total: 123.74016 MB Change: Increased, 1617.92 KB, 1.325 %
Dec 10 11:32:34 - MainThread - Memory usage: Total: 123.744256 MB Change: Increased, 4.096 KB, 0.003 %
Dec 10 11:33:08 - MainThread - Memory usage: Total: 124.141568 MB Change: Increased, 397.312 KB, 0.321 %
Dec 10 11:33:42 - MainThread - Memory usage: Total: 125.640704 MB Change: Increased, 1499.136 KB, 1.208 %
Dec 10 11:33:55 - MainThread - Memory usage: Total: 127.61088 MB Change: Increased, 1970.176 KB, 1.568 %
Dec 10 11:34:08 - MainThread - Memory usage: Total: 128.069632 MB Change: Increased, 458.752 KB, 0.359 %
Dec 10 11:34:42 - MainThread - Memory usage: Total: 128.323584 MB Change: Increased, 253.952 KB, 0.198 %
Dec 10 11:34:56 - MainThread - Memory usage: Total: 129.032192 MB Change: Increased, 708.608 KB, 0.552 %
Dec 10 11:35:19 - MainThread - Memory usage: Total: 129.396736 MB Change: Increased, 364.544 KB, 0.283 %
Dec 10 11:35:33 - MainThread - Memory usage: Total: 129.396736 MB Change: No change, 0.0 KB, 0.000 %
Dec 10 11:35:57 - MainThread - Memory usage: Total: 129.568768 MB Change: Increased, 172.032 KB, 0.133 %
Dec 10 11:36:30 - MainThread - Memory usage: Total: 129.744896 MB Change: Increased, 176.128 KB, 0.136 %
Dec 10 11:36:44 - MainThread - Memory usage: Total: 129.916928 MB Change: Increased, 172.032 KB, 0.133 %
Dec 10 11:37:08 - MainThread - Memory usage: Total: 129.982464 MB Change: Increased, 65.536 KB, 0.050 %
Dec 10 11:37:21 - MainThread - Memory usage: Total: 130.33472 MB Change: Increased, 352.256 KB, 0.271 %
Dec 10 11:37:34 - MainThread - Memory usage: Total: 130.781184 MB Change: Increased, 446.464 KB, 0.343 %
Dec 10 11:38:04 - MainThread - Memory usage: Total: 131.436544 MB Change: Increased, 655.36 KB, 0.501 %
Dec 10 11:38:47 - MainThread - Memory usage: Total: 131.80928 MB Change: Increased, 372.736 KB, 0.284 %
Dec 10 11:39:21 - MainThread - Memory usage: Total: 131.82976 MB Change: Increased, 20.48 KB, 0.016 %
Dec 10 11:39:55 - MainThread - Memory usage: Total: 131.85024 MB Change: Increased, 20.48 KB, 0.016 %
Dec 10 11:40:18 - MainThread - Memory usage: Total: 131.82976 MB Change: Decreased, -20.48 KB, -0.016 %
Dec 10 11:40:42 - MainThread - Memory usage: Total: 130.7648 MB Change: Decreased, -1064.96 KB, -0.808 %
Dec 10 11:41:06 - MainThread - Memory usage: Total: 131.022848 MB Change: Increased, 258.048 KB, 0.197 %
Dec 10 11:41:29 - MainThread - Memory usage: Total: 131.141632 MB Change: Increased, 118.784 KB, 0.091 %
Dec 10 11:42:03 - MainThread - Memory usage: Total: 133.20192 MB Change: Increased, 2060.288 KB, 1.571 %
Dec 10 11:42:27 - MainThread - Memory usage: Total: 133.353472 MB Change: Increased, 151.552 KB, 0.114 %
Dec 10 11:43:10 - MainThread - Memory usage: Total: 134.520832 MB Change: Increased, 1167.36 KB, 0.875 %
Dec 10 11:43:24 - MainThread - Memory usage: Total: 134.844416 MB Change: Increased, 323.584 KB, 0.241 %
Dec 10 11:43:37 - MainThread - Memory usage: Total: 134.606848 MB Change: Decreased, -237.568 KB, -0.176 %
Dec 10 11:44:11 - MainThread - Memory usage: Total: 134.47168 MB Change: Decreased, -135.168 KB, -0.100 %
Dec 10 11:44:45 - MainThread - Memory usage: Total: 134.823936 MB Change: Increased, 352.256 KB, 0.262 %
Dec 10 11:45:08 - MainThread - Memory usage: Total: 134.590464 MB Change: Decreased, -233.472 KB, -0.173 %
Dec 10 11:45:22 - MainThread - Memory usage: Total: 134.782976 MB Change: Increased, 192.512 KB, 0.143 %
Dec 10 11:45:33 - MainThread - Memory usage: Total: 132.472832 MB Change: Decreased, -2310.144 KB, -1.714 %
Dec 10 11:45:57 - MainThread - Memory usage: Total: 134.086656 MB Change: Increased, 1613.824 KB, 1.218 %
Dec 10 11:46:21 - MainThread - Memory usage: Total: 134.094848 MB Change: Increased, 8.192 KB, 0.006 %
Dec 10 11:46:55 - MainThread - Memory usage: Total: 133.730304 MB Change: Decreased, -364.544 KB, -0.272 %
Dec 10 11:47:38 - MainThread - Memory usage: Total: 133.726208 MB Change: Decreased, -4.096 KB, -0.003 %
Dec 10 11:48:12 - MainThread - Memory usage: Total: 134.238208 MB Change: Increased, 512.0 KB, 0.383 %
Dec 10 11:48:46 - MainThread - Memory usage: Total: 134.467584 MB Change: Increased, 229.376 KB, 0.171 %
Dec 10 11:48:59 - MainThread - Memory usage: Total: 134.545408 MB Change: Increased, 77.824 KB, 0.058 %
Dec 10 11:49:42 - MainThread - Memory usage: Total: 134.631424 MB Change: Increased, 86.016 KB, 0.064 %
Dec 10 11:50:06 - MainThread - Memory usage: Total: 134.729728 MB Change: Increased, 98.304 KB, 0.073 %
Dec 10 11:50:19 - MainThread - Memory usage: Total: 134.713344 MB Change: Decreased, -16.384 KB, -0.012 %
Dec 10 11:50:53 - MainThread - Memory usage: Total: 134.619136 MB Change: Decreased, -94.208 KB, -0.070 %
Dec 10 11:51:27 - MainThread - Memory usage: Total: 134.627328 MB Change: Increased, 8.192 KB, 0.006 %
Dec 10 11:52:10 - MainThread - Memory usage: Total: 134.361088 MB Change: Decreased, -266.24 KB, -0.198 %
Dec 10 11:52:34 - MainThread - Memory usage: Total: 134.524928 MB Change: Increased, 163.84 KB, 0.122 %
Dec 10 11:53:17 - MainThread - Memory usage: Total: 134.38976 MB Change: Decreased, -135.168 KB, -0.100 %
Dec 10 11:53:31 - MainThread - Memory usage: Total: 134.733824 MB Change: Increased, 344.064 KB, 0.256 %
Dec 10 11:53:44 - MainThread - Memory usage: Total: 135.20896 MB Change: Increased, 475.136 KB, 0.353 %
Dec 10 11:54:18 - MainThread - Memory usage: Total: 135.266304 MB Change: Increased, 57.344 KB, 0.042 %
Dec 10 11:54:31 - MainThread - Memory usage: Total: 134.275072 MB Change: Decreased, -991.232 KB, -0.733 %
Dec 10 11:55:15 - MainThread - Memory usage: Total: 135.348224 MB Change: Increased, 1073.152 KB, 0.799 %
Dec 10 11:55:39 - MainThread - Memory usage: Total: 135.438336 MB Change: Increased, 90.112 KB, 0.067 %
Dec 10 11:56:22 - MainThread - Memory usage: Total: 135.507968 MB Change: Increased, 69.632 KB, 0.051 %
Dec 10 11:56:56 - MainThread - Memory usage: Total: 135.524352 MB Change: Increased, 16.384 KB, 0.012 %
Dec 10 11:57:39 - MainThread - Memory usage: Total: 135.3728 MB Change: Decreased, -151.552 KB, -0.112 %
Dec 10 11:57:57 - MainThread - Memory usage: Total: 135.55712 MB Change: Increased, 184.32 KB, 0.136 %
Dec 10 11:58:40 - MainThread - Memory usage: Total: 135.467008 MB Change: Decreased, -90.112 KB, -0.066 %
Dec 10 11:59:14 - MainThread - Memory usage: Total: 135.63904 MB Change: Increased, 172.032 KB, 0.127 %
Dec 10 11:59:48 - MainThread - Memory usage: Total: 135.995392 MB Change: Increased, 356.352 KB, 0.263 %
Dec 10 12:00:07 - MainThread - Memory usage: Total: 136.122368 MB Change: Increased, 126.976 KB, 0.093 %
Dec 10 12:00:20 - MainThread - Memory usage: Total: 135.974912 MB Change: Decreased, -147.456 KB, -0.108 %
Dec 10 12:00:54 - MainThread - Memory usage: Total: 135.667712 MB Change: Decreased, -307.2 KB, -0.226 %
Dec 10 12:01:28 - MainThread - Memory usage: Total: 136.093696 MB Change: Increased, 425.984 KB, 0.314 %
Dec 10 12:01:52 - MainThread - Memory usage: Total: 134.971392 MB Change: Decreased, -1122.304 KB, -0.825 %
Dec 10 12:02:05 - MainThread - Memory usage: Total: 136.00768 MB Change: Increased, 1036.288 KB, 0.768 %
Dec 10 12:02:48 - MainThread - Memory usage: Total: 135.282688 MB Change: Decreased, -724.992 KB, -0.533 %
Dec 10 12:03:22 - MainThread - Memory usage: Total: 136.085504 MB Change: Increased, 802.816 KB, 0.593 %
Dec 10 12:03:46 - MainThread - Memory usage: Total: 135.806976 MB Change: Decreased, -278.528 KB, -0.205 %
Dec 10 12:04:29 - MainThread - Memory usage: Total: 136.585216 MB Change: Increased, 778.24 KB, 0.573 %
Dec 10 12:04:42 - MainThread - Memory usage: Total: 135.479296 MB Change: Decreased, -1105.92 KB, -0.810 %
Dec 10 12:05:16 - MainThread - Memory usage: Total: 136.364032 MB Change: Increased, 884.736 KB, 0.653 %
Dec 10 12:05:50 - MainThread - Memory usage: Total: 136.368128 MB Change: Increased, 4.096 KB, 0.003 %
Dec 10 12:06:23 - MainThread - Memory usage: Total: 137.043968 MB Change: Increased, 675.84 KB, 0.496 %
Dec 10 12:06:47 - MainThread - Memory usage: Total: 137.224192 MB Change: Increased, 180.224 KB, 0.132 %
Dec 10 12:07:31 - MainThread - Memory usage: Total: 137.224192 MB Change: No change, 0.0 KB, 0.000 %
Dec 10 12:07:55 - MainThread - Memory usage: Total: 137.3184 MB Change: Increased, 94.208 KB, 0.069 %
Dec 10 12:08:29 - MainThread - Memory usage: Total: 137.515008 MB Change: Increased, 196.608 KB, 0.143 %
Dec 10 12:08:52 - MainThread - Memory usage: Total: 137.609216 MB Change: Increased, 94.208 KB, 0.069 %
Dec 10 12:09:26 - MainThread - Memory usage: Total: 137.150464 MB Change: Decreased, -458.752 KB, -0.333 %
Dec 10 12:09:51 - MainThread - Memory usage: Total: 137.662464 MB Change: Increased, 512.0 KB, 0.373 %
Dec 10 12:10:03 - MainThread - Memory usage: Total: 136.843264 MB Change: Decreased, -819.2 KB, -0.595 %
Dec 10 12:10:37 - MainThread - Memory usage: Total: 137.814016 MB Change: Increased, 970.752 KB, 0.709 %
Dec 10 12:10:50 - MainThread - Memory usage: Total: 137.822208 MB Change: Increased, 8.192 KB, 0.006 %
Dec 10 12:11:33 - MainThread - Memory usage: Total: 138.080256 MB Change: Increased, 258.048 KB, 0.187 %
Dec 10 12:12:07 - MainThread - Memory usage: Total: 137.551872 MB Change: Decreased, -528.384 KB, -0.383 %
Dec 10 12:12:20 - MainThread - Memory usage: Total: 137.003008 MB Change: Decreased, -548.864 KB, -0.399 %
Dec 10 12:13:04 - MainThread - Memory usage: Total: 137.433088 MB Change: Increased, 430.08 KB, 0.314 %
Dec 10 12:13:28 - MainThread - Memory usage: Total: 137.48224 MB Change: Increased, 49.152 KB, 0.036 %
Dec 10 12:14:11 - MainThread - Memory usage: Total: 138.190848 MB Change: Increased, 708.608 KB, 0.515 %
Dec 10 12:14:35 - MainThread - Memory usage: Total: 138.387456 MB Change: Increased, 196.608 KB, 0.142 %
Dec 10 12:15:09 - MainThread - Memory usage: Total: 138.682368 MB Change: Increased, 294.912 KB, 0.213 %
Dec 10 12:15:42 - MainThread - Memory usage: Total: 138.858496 MB Change: Increased, 176.128 KB, 0.127 %
Dec 10 12:16:16 - MainThread - Memory usage: Total: 138.616832 MB Change: Decreased, -241.664 KB, -0.174 %
Dec 10 12:16:29 - MainThread - Memory usage: Total: 138.924032 MB Change: Increased, 307.2 KB, 0.222 %
Dec 10 12:17:03 - MainThread - Memory usage: Total: 138.81344 MB Change: Decreased, -110.592 KB, -0.080 %
Dec 10 12:17:16 - MainThread - Memory usage: Total: 138.346496 MB Change: Decreased, -466.944 KB, -0.336 %
Dec 10 12:17:50 - MainThread - Memory usage: Total: 139.051008 MB Change: Increased, 704.512 KB, 0.509 %
Dec 10 12:18:23 - MainThread - Memory usage: Total: 139.10016 MB Change: Increased, 49.152 KB, 0.035 %

Can someone suggest how to properly cleanup memory to avoid this memory leak?

Answer 1

Answering my own question.

Please see the discussion here https://github.com/pycurl/pycurl/issues/721

Apparently PyCurl does not play well with Python multi-threading when thread count is large enough, but works fine in single thread. For scenarios requiring concurrent API calls it is recommended to use pycurl.CurlMulti

Recommendation: https://github.com/pycurl/pycurl/issues/721#issuecomment-992632719

Test Code: https://github.com/pycurl/pycurl/issues/721#issuecomment-994475703