Object of type TemporaryUploadedFile is not JSON serializable

Question

I am trying to run my file upload view in Django as a task using celery, every works fine except when the image file is more than 2.5MB which celery complains that it cant serialize. the reason i am using celery is that I want to get the progress info from each file upload task and create a progress bar from it. below is my code.

/views.py
def simple_upload(request):
    if request.method == 'POST' and request.FILES['image']:
        file = request.FILES['image']
        #process_download(file)
        task = process_download.delay(file)
        #print(task.task_id)
        return redirect('index')
    return render(request, 'upload.html')

@shared_task(bind=True)
def process_download(self, image_file):
    process_recoder = ProgressRecorder(self)
    print('Upload: Task Started')
    # time.sleep(50)
    fs = FileSystemStorage()
    buffer = io.BytesIO()
    chunk_size = 0
    for chunk in image_file.chunks():
        chunk_size += sys.getsizeof(chunk)
        buffer.write(chunk)
        process_recoder.set_progress(chunk_size, image_file.size)
    buffer.seek(0)
    image = ImageFile(buffer, name=image_file.name)
    fs.save(image_file.name, image)
    return 'Done'

is there any way i could make celery serialize the images that are larger than 2.5, using the Django settings FILE_UPLOAD_MAX_MEMORY_SIZE = 50*1024*1024 does not work too?

Answer 1

You could store this image file inside a Model of your application. You can then pass this model's id field to the Celery task - which is guaranteed to work.

Inside the Celery task, you take care of fetching the image file from the model, and then doing your stuff.

Your view would become:

def simple_upload(request):
    if request.method == 'POST' and request.FILES['image']:
        file = request.FILES['image']
        image_upload = ImageUpload.objects.create(file=file)
        task = process_download.delay(image_upload.id)
        return redirect('index')
    return render(request, 'upload.html')

And your celery task:

@shared_task(bind=True)
def process_download(self, image_upload_id):
    image_file = ImageUpload.objects.get(id=image_upload_id).file

    process_recoder = ProgressRecorder(self)
    print('Upload: Task Started')
    # time.sleep(50)
    fs = FileSystemStorage()
    buffer = io.BytesIO()
    chunk_size = 0
    for chunk in image_file.chunks():
        chunk_size += sys.getsizeof(chunk)
        buffer.write(chunk)
        process_recoder.set_progress(chunk_size, image_file.size)
    buffer.seek(0)
    image = ImageFile(buffer, name=image_file.name)
    fs.save(image_file.name, image)
    return 'Done'