Modify DRF Request objects for dispatch to other views

Question

Context

AWS Elastic Beanstalk Worker tiers can be configured to accept a single AWS SQS queue automatically. Incoming messages on this queue can be routed to a single endpoint which some instance in the worker tier will respond to; they show up to the workers as POST events.

The software my workers are running is based on Django/DRF.

Goal

I want my worker tier instances to be able to handle more than one type of incoming event. Ideally, SQS could be configured to deliver the requests to more than one endpoint, but that appears to be impossible.

My implementation

class DispatchSerializer(serializers.Serializer):
    "`to` is the name of the destination endpoint; `payload` is the data I want delivered"
    to = serializers.CharField(
        max_length=80,
    )

    payload = serializers.JSONField()

@api_view(http_method_names=['POST'])
@permission_classes([AllowAny])
def dispatch(request):
    """
    Dispatch a request to a sibling endpoint.
    """
    routing_table = {
        ... # this is specific to my application; 
            # it's just a dict of names to views
    }

    serializer = DispatchSerializer(data=request.data)
    if serializer.is_valid(raise_exception=True):
        to = serializer.validated_data['to']
        try:
            view = routing_table[to]
        except KeyError:
            raise Http404()

        # https://github.com/encode/django-rest-framework/blob/master/rest_framework/request.py#L183-L187
        request._full_data = serializer.validated_data['payload']
        return view(request)

    # If this returns other than 200, SQS simply re-attempts the request later. 
    # No returned data is preserved, so we might as well not return any.
    return Response()

As you can see, I attempt to simply replace _full_data attribute of the request with the payload, so that the inner view sees only the data intended for it (which to the dispatch view is the payload).

The problem

This implementation doesn't actually work. I get errors of this form:

Traceback (most recent call last):
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 378, in __getattribute__
    return super(Request, self).__getattribute__(attr)
AttributeError: 'Request' object has no attribute 'body'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 378, in __getattribute__
    return super(Request, self).__getattribute__(attr)
AttributeError: 'Request' object has no attribute 'body'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/views.py", line 483, in dispatch
    response = self.handle_exception(exc)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/views.py", line 443, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/decorators.py", line 52, in handler
    return func(*args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/project/django-backend/cardamom/worker/views.py", line 196, in dispatch
    return view(request)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/views.py", line 483, in dispatch
    response = self.handle_exception(exc)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/views.py", line 443, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/decorators.py", line 52, in handler
    return func(*args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/project/django-backend/cardamom/core/views/mailchimp.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/project/django-backend/cardamom/worker/views.py", line 52, in wrapper
    result = f(log_entry, *args, **kwargs)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/project/django-backend/cardamom/worker/views.py", line 106, in match_one
    user_id = int(request.data['user_id'])
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 378, in __getattribute__
    return super(Request, self).__getattribute__(attr)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 186, in data
    self._load_data_and_files()
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 246, in _load_data_and_files
    self._data, self._files = self._parse()
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 290, in _parse
    stream = self.stream
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 378, in __getattribute__
    return super(Request, self).__getattribute__(attr)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 173, in stream
    self._load_stream()
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 270, in _load_stream
    self._stream = six.BytesIO(self.body)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 382, in __getattribute__
    return getattr(self._request, attr)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/rest_framework/request.py", line 382, in __getattribute__
    return getattr(self._request, attr)
  File "/mnt/d/Users/coriolinus/Documents/Toptal/cardamom/.venv/lib/python3.4/site-packages/django/http/request.py", line 264, in body
    raise RawPostDataException("You cannot access body after reading from request's data stream")
django.http.request.RawPostDataException: You cannot access body after reading from request's data stream

Problem restatement

The request object is complicated; generating a new one is implausible. At the same time, we need to modify the data of this one in order to properly pass it along to the next view, so that view only sees the data in its expected format.

How can I accomplish this?

[edit] Passing an un-modified request doesn't work either

I tried editing the above example to remove any modification of the request object, and instead created the following serializer mixin:

class UndispatcherMixin:
    """
    Adjust a serializer such that it can accept its normal serialization,
    or alternately the payload of a DispatchSerializer.
    """

    def is_valid(self, raise_exception=False):
        if not hasattr(self, '_validated_data'):
            assert hasattr(self, 'initial_data'), (
                'Cannot call `.is_valid()` as no `data=` keyword argument was '
                'passed when instantiating the serializer instance.'
            )
            ds = DispatchSerializer(data=self.initial_data)

            if ds.is_valid(raise_exception=False):
                self.initial_data = ds.validated_data['payload']
        return super().is_valid(raise_exception=raise_exception)

This mixin was then added to the relevant serializers used by the downstream views. Interestingly, it failed with the same django.http.RawPostDataException as before.

It's received wisdom all over SO that you can just call your views as normal functions, so long as you pass in a proper request object. This may not actually be true, at least for the case where the view was created by the DRF @api_view decorator.

Still working on a way to actually solve this.

Answer 1

For whatever reason--this may be an error in DRF, but I'm not sure--you can't just call an @api_view function from within another one and pass along the request object. It just doesn't work.

What does work is to factor out the code which does the actual work, and call it separately from the @api_view function and also the dispatch view. That is, something like this:

def perform_action(request, data):
    """
    Do something

    `request` is included for completeness in case it's necessary,
    but the data normally read from `request.data` should be read instead
    from the `data` argument.
    """
    serializer = ActionSerializer(data=data)
    if serializer.is_valid():
        ... # Whatever your action should be, goes here

@api_view(http_method_names=['POST'])
@permission_classes([AllowAny])
def action(request)
    perform_action(request, request.data)
    return Response()  # 200, no content

@api_view(http_method_names=['POST'])
@permission_classes([AllowAny])
def dispatch(request):
    """
    Dispatch a request to a sibling endpoint.
    """
    routing_table = {
        'action': perform_action, # etc...
    }

    serializer = DispatchSerializer(data=request.data)
    if serializer.is_valid(raise_exception=True):
        to = serializer.validated_data['to']
        try:
            view = routing_table[to]
        except KeyError:
            raise Http404()

        view(request, serializer.validated_data['payload'])

    return Response()