Custom json serialization in celery

Question

I am trying to use celery with custom objects for which I have implemented a custom serializer, but the celery workers try to use pickling.

celeryconfig.py

broker_url = 'redis://localhost'
result_backend = 'redis://localhost'
imports = ('tasks',)
accept_content = ['application/x-json']
task_serializer = 'custom_json'
result_serializer = 'custom_json'

app.py

from celery import Celery
from . import serializers
from . import celeryconfig

from kombu import serialization
serialization.register(
    'custom_json',
    serializers.dumps,
    serializers.loads,
    content_type='application/x-json',
    content_encoding='utf-8',
)

app = Celery()
app.config_from_object(celeryconfig)

if __name__ == '__main__':
    app.start()

main.py

from .tasks import my_task
my_obj = CustomClass()
my_task.delay(my_obj)

This code works fine if my class was defined in python:

class CustomClass:
    def __init__(self):
        ...

But my CustomClass actually comes from a Boost.Python binding, that I import from an.so file, and then I get the following error from the worker:

[2020-04-11 16:25:08,102: INFO/MainProcess] Received task: my_task[f73a3119-65d7-4a04-9e0d-2bc25ad19dde]  
...
RuntimeError: Pickling of "CustomClass" instances is not enabled (http://www.boost.org/libs/python/doc/v2/pickle.html)

I understand that the error message suggests to dig into their pickle's specific. But the whole point of using custom json serializers is to not go down this road.

So my question is: why is celery even trying to use pickling?

Edit: As a dummy example, the following class (without boost) that is not picklable would yield the following error:

 Unrecoverable error: TypeError("can't pickle generator objects",)

class NonPicklableClass:

    def __init__(self, arg):
        self.gen = (i for i in arg)


class CustomEncoder(json.JSONEncoder):

    def default(self, o):
        if isinstance(o, NonPicklableClass):
            return {
                '__type__': 'custom',
                'raw': list(o.gen),
            }
        return o

def hook(o):
    dtype = o.get('__type__')
    if dtype == 'custom':
        return NonPicklableClass(o['raw'])

def dumps(o):
    return json.dumps(o, cls=CustomEncoder)

def loads(s):
    return json.loads(s, object_hook=hook)

I clearly must be misunderstanding something

Answer 1

I think I figured it out, the jobs are sent to the workers using the custom serializers.

However within each worker, the data is passed through each process using the regular python pickling.