OSError: [Errno 24] 通过 Django admin 上传 9000+ csv 个文件时打开的文件太多

Question

I've been struggling to upload csv files containing data to populate the database of my Django project. I am serving my project using Django 3.1 and Gunicorn if that is important. I've referred to multiple stack overflow posts regarding the same issue, but none of them have resolved this problem. Just to list out the steps I've taken to solve this problem:

• Ran ulimit -n 50000 in terminal to increase the max number of files that can be opened as suggested by this post .
• Used ulimit -n to find the max number of files that can be opened and changed the configurations of this limit in the limits.conf file, as suggested by this post , to changed the system limits. I confirmed it was changed by running ulimit -a
• I believed it might be a memory issue and that Linux was limiting the max amount of space available on the heap so I changhed the configuration for that as well. Fortunately, it was not a memory issue since RAM usage appeared to be extremely stable according to my control panel

Here is the code related to my issue:

admin.py

class CsvImportForm(forms.Form):
    csv_upload = forms.FileField(widget=forms.ClearableFileInput(attrs={'multiple': True}))

@admin.register(dummyClass)
class dummyClassAdmin(admin.ModelAdmin):
    search_fields = search fields
    list_display = stuff to display in a list
    list_filter = thing that filters objects
    prepopulated_fields = prepopulated field

@csrf_exempt
def get_urls(self):
    urls = super().get_urls()
    new_urls = [path('upload-csv/', self.upload_csv),]
    return new_urls + urls

@csrf_exempt
def upload_epcsv(self, request):

    if request.method == "POST":
        csv_files = request.FILES.getlist('csv_upload')
        errors = []
        for csv_file in csv_files:
            if not(csv_file.name.endswith('.csv') or  csv_file.name.endswith('.xlsx')):
                messages.warning(request, 'The wrong file type was uploaded')
                return HttpResponseRedirect(request.path_info)

            csv_data = None
            if csv_file.name.endswith('.xlsx'):
                csv_data = pd.read_excel(csv_file, sheet_name=0, engine='openpyxl')
            else:
                csv_data = pd.read_csv(csv_file)
            csv_data.fillna("https://www.a.com", inplace=True)
        
            
            *Insert code that parses through dataframe and populates fields in database with data



        url = reverse('admin:index')
        return HttpResponseRedirect(url)

    form = CsvImportForm()
    data = {"form": form}
    return render(request, "admin/csv_upload.html", data)

Here is the traceback of the error:

Traceback (most recent call last):
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
  File "/path_of_venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
  File "/path_of_app/admin.py", line 185, in upload_epcsv
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/wsgi.py", line 116, in FILES
  File "/path_of_venv/lib/python3.8/site-packages/django/http/request.py", line 350, in _load_post_and_files
  File "/path_of_venv/lib/python3.8/site-packages/django/http/request.py", line 310, in parse_file_upload
  File "/path_of_venv/lib/python3.8/site-packages/django/http/multipartparser.py", line 231, in parse
  File "/path_of_venv/lib/python3.8/site-packages/django/core/files/uploadhandler.py", line 140, in new_file
  File "/path_of_venv/lib/python3.8/site-packages/django/core/files/uploadedfile.py", line 61, in __init__
  File "/.pyenv/versions/3.8.3/lib/python3.8/tempfile.py", line 541, in NamedTemporaryFile
  File "/.pyenv/versions/3.8.3/lib/python3.8/tempfile.py", line 250, in _mkstemp_inner
OSError: [Errno 24] Too many open files: '/tmp/tmpgnj0th91.upload.csv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/utils/deprecation.py", line 114, in __call__
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 49, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 52, in technical_500_response
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 328, in get_traceback_html
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1218, in open
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1074, in _opener
OSError: [Errno 24] Too many open files: '/path_of_venv/lib/python3.8/site-packages/django/views/templates/technical_500.html'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/utils/deprecation.py", line 114, in __call__
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 49, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 52, in technical_500_response
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 328, in get_traceback_html
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1218, in open
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1074, in _opener
OSError: [Errno 24] Too many open files: '/path_of_venv/lib/python3.8/site-packages/django/views/templates/technical_500.html'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/utils/deprecation.py", line 114, in __call__
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 49, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 52, in technical_500_response
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 328, in get_traceback_html
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1218, in open
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1074, in _opener
OSError: [Errno 24] Too many open files: '/path_of_venv/django/views/templates/technical_500.html'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/utils/deprecation.py", line 114, in __call__
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 49, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 52, in technical_500_response
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 328, in get_traceback_html
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1218, in open
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1074, in _opener
OSError: [Errno 24] Too many open files: '/path_of_venv/django/views/templates/technical_500.html'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path_of_venv/lib/python3.8/site-packages/gevent/pywsgi.py", line 999, in handle_one_response
  File "/path_of_venv/lib/python3.8/site-packages/geventwebsocket/handler.py", line 87, in run_application
  File "/path_of_venv/lib/python3.8/site-packages/gevent/pywsgi.py", line 945, in run_application
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/wsgi.py", line 133, in __call__
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 130, in get_response
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 49, in inner
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 103, in response_for_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 138, in handle_uncaught_exception
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 52, in technical_500_response
  File "/path_of_venv/lib/python3.8/site-packages/django/views/debug.py", line 328, in get_traceback_html
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1218, in open
  File "/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py", line 1074, in _opener
OSError: [Errno 24] Too many open files: '/path_of_venv/lib/python3.8/site-packages/django/views/templates/technical_500.html'
2022-03-27T17:21:27Z {'REMOTE_ADDR': '127.0.0.1', 'REMOTE_PORT': '55555', 'HTTP_HOST': 'mysite.com', (hidden keys: 40)} failed with OSError

Changed the path names for privacy reasons

Answer 1

It seems that is problem is pd.read_csv(csv_file) won't close csv_file which type is UploadedFile . Those open files never close and leaked as a result.

You can try to log csv_data.closed to see if it's still open.

Solutions:

1. (Recommend) Use temporary_file_path while reading it, it will close the file after reading.
2. Close it by yourself csv_file.close() finally.