How to run Faust from Docker - ERROR: Failed building wheel for python-rocksdb

Question

I'm trying to run Python Faust from Docker.

Based on this documentation: https://faust.readthedocs.io/en/latest/userguide/installation.html

I created a simple Docker file:

FROM python:3

ADD ./app/app.py /

RUN pip3 install --upgrade pip

RUN pip install -U faust

RUN pip install "faust[rocksdb]"

RUN pip install "faust[rocksdb,uvloop,fast,redis]"

CMD ["python", "./app.py"]

When I create a docker file I receive an error at the 5th stage (Step 5/7: RUN pip install "faust[rocksdb]")

---> Running in 1e42a5e50cbe Requirement already satisfied: faust[rocksdb] in /usr/local/lib/python3.10/site-packages (1.10.4) Requirement already satisfied: terminaltables<4.0,>=3.1 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (3.1.10) Requirement already satisfied: click<8.0,>=6.7 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (7.1.2) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.7.2) Requirement already satisfied: aiohttp-cors<2.0,>=0.7 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (0.7.0) Requirement already satisfied: mypy-extensions in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (0.4.3) Requirement already satisfied: colorclass<3.0,>=2.2 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (2.2.2) Requirement already satisfied: opentracing<2.0.0,>=1.3.0 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.3.0) Re quirement already satisfied: mode<4.4,>=4.3.2 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (4.3.2) Requirement already satisfied: venusian<2.0,>=1.1 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.2.0) Requirement already satisfied: aiohttp<4.0,>=3.5.2 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (3.8.1) Requirement already satisfied: robinhood-aiokafka<1.2,>=1.1.6 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.1.6) Requirement already satisfied: croniter>=0.3.16 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.1.0) Collecting python-rocksdb>=0.6.7 Downloading python-rocksdb-0.7.0.tar.gz (219 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (1.2.0) Requirement already satisfied: attrs>=17.3.0 in /usr/loc al/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (21.2.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (1.2.0) Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (2.0.9) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (5.2.0) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (4.0.2) Requirement already satisfied: python-dateutil in /usr/local/lib/python3.10/site-packages (from croniter>=0.3.16->faust[rocksdb]) (2.8.2) Requirement already satisfied: colorlog>=2.9.0 in /usr/local/lib/python3.10/site-packages (from mode<4.4,>=4.3.2->faust[rocksdb]) (6.6.0) Requirement already satisfied: setuptools>=25 in /usr/local/lib/python3.10/site-packages (from python-rocksdb>=0.6.7->faust[rocksdb]) (57.5.0) Requirement already satisfied: kafka-python<1.5,>=1.4.6 in /usr/local/lib/python3.10/site-packages (from robinhood-aiokafka<1.2,>=1.1.6->faust[rocksdb]) (1.4.7) Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.10/site-packages (from yarl<2.0,>=1.0->faust[rocksdb]) (3.3) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/site-packages (from python-dateutil->croniter>=0.3.16->faust[rocksdb]) (1.16.0)

And an ERROR PART :

Building wheels for collected packages: python-rocksdb Building wheel for python-rocksdb (setup.py): started ERROR: Command errored out with exit status 1: command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/setup.py'"'"'; file ='"'"'/tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)( file ) if os.path.exists( file ) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file , '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-9_o4ek6z cwd: /tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/ Complete output (64 lines): running bdist_wheel running build
running build_py creating build creating build/lib.linux-x86_64-3.10 creating build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/interfaces.py -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/errors.py -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/merge_operators.py -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/ init .py -> build/lib.linux-x86_64-3.10/rocksdb
creating build/lib.linux-x86_64-3.10/rocksdb/tests copying rocksdb/tests/test_memtable.py -> build/lib.linux-x86_64-3.10/rocksdb/tests copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-3.10/rocksdb/tests copying rocksdb/tests/ init .py -> build/lib.linux-x86_64-3.10/rocksdb/tests copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-3.10/rocksdb/tests running egg_info writing python_rocksdb.egg-info/PKG-INFO writing dependency_links to python_rocksdb.egg-info/dependency_links.txt writing requirements to python_rocksdb.egg-info/requires.txt writing top-level names to python_rocksdb.egg-info/top_level.txt reading manifest file 'python_rocksdb.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'python_rocksdb.egg-info/SOURCES.txt' copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/ rocksdb.pyx -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/backup.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/cache.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/comparator.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/db.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/env.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/filter_policy.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/iterator.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/logger.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/merge_operator.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/options.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/slice .pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/slice_transform.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/snapshot.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/status.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/std_memory.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/table_factory.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/universal_compaction.pxd -> build/lib.linux-x86_64-3.10/rocksdb creating build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/comparator_wrapper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/memtable_factories.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/utils.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp
copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp running build_ext
cythoning rocksdb/_rocksdb.pyx to rocksdb/_rocksdb.cpp
/tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/.eggs/Cython-0.29.26-py3.10-linux-x86_64.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release: File. /tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/rocksdb/_rocksdb.pyx tree = Parsing,p_module(s, pxd. full_module_name) building 'rocksdb._rocksdb' extension creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10/rocksdb gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.10 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-3.10/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing -fno-rtti rocksdb/_rocksdb.cpp:705:10: fatal error: rocksdb/slice.h: No such file or directory 705 | #include "rocksdb/slice.h" | ^~~~~~~~~~~~~~~~~ compilation terminated. error: command '/usr/bin/gcc' failed with exit code 1
---------------------------------------- Building wheel for python-rocksdb (setup.py): finished with status 'error' ERROR: Failed building wheel for python-rocksdb

Can anyone help me to move on with this? I'd like to use Faust from Docker on Kubernetes.

Answer 1

Read the error message, where it is clearly stated you are missing a header file:

fatal error: rocksdb/slice.h: No such file or directory 705 | #include "rocksdb/slice.h" | ^~~~~~~~~~~~~~~~~ compilation terminated. error: command '/usr/bin/gcc' failed with exit code 1

Accordingly, you'll need to build and install RocksDB. This is separate from the installation of faust[rocksdb] with pip. That simply installs python-rocksdb , the Python interface to the underlying libraries.

There is even a (third-party) RocksDB docker image based on Python 3.7 Slim .

You could use that directly or take some tricks from the Dockerfile for that image.