django-pipeline not working with S3Boto3Storage

Question

How can I use S3Boto3Storage with django-pipeline?

According to the docs it looks like it would work: https://django-pipeline.readthedocs.io/en/latest/storages.html#using-with-other-storages

The documentation looks a little bit outdated, because the package name and class name changed over the past years.

This is how my storage class looks like with the correct package and class name.

from django.contrib.staticfiles.storage import ManifestFilesMixin
from pipeline.storage import PipelineMixin
from storages.backends.s3boto3 import S3Boto3Storage

class S3PipelineManifestStorage(PipelineMixin, ManifestFilesMixin, S3Boto3Storage):
    pass

In my settings.py I set the Storage class to the above created one.

STATICFILES_STORAGE = 'myproject.storages.S3PipelineManifestStorage'

running collectstatic is returning this two errors

python3 manage.py collectstatic
This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
Traceback (most recent call last):
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/storages/backends/s3boto3.py", line 430, in _open
    f = S3Boto3StorageFile(name, mode, self)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/storages/backends/s3boto3.py", line 111, in __init__
    self.obj.load()
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/boto3/resources/factory.py", line 564, in do_action
    response = action(self, *args, **kwargs)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 205, in call_if_sampling
    return wrapped(*args, **kwargs)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 187, in handle
    collected = self.collect()
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 128, in collect
    for original_path, processed_path, processed in processor:
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/pipeline/storage.py", line 30, in post_process
    packager.pack_stylesheets(package)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/pipeline/packager.py", line 98, in pack_stylesheets
    variant=package.variant, **kwargs)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/pipeline/packager.py", line 116, in pack
    content = compress(paths, **kwargs)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/pipeline/compressors/__init__.py", line 72, in compress_css
    css = self.concatenate_and_rewrite(paths, output_filename, variant)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/pipeline/compressors/__init__.py", line 136, in concatenate_and_rewrite
    content = self.read_text(path)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/pipeline/compressors/__init__.py", line 219, in read_text
    content = self.read_bytes(path)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/pipeline/compressors/__init__.py", line 213, in read_bytes
    file = staticfiles_storage.open(path)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/django/core/files/storage.py", line 38, in open
    return self._open(name, mode)
  File "/home/myproject/git/myvenv/lib/python3.7/site-packages/storages/backends/s3boto3.py", line 433, in _open
    raise FileNotFoundError('File does not exist: %s' % name)
FileNotFoundError: File does not exist: static/scss/adminmain.css

1. Any idea what is wrong?
2. Was someone able to use this combination succefully or is this feature broken in 2022?

Answer 1

With the current master branch it works perfectly.

pip3 install git+https://github.com/jazzband/django-pipeline.git@c85a2572e53218716359126d9465cc63447817e5

There is a bug in the current 2.0.8 version. So wait for the next release.