AWS Elastic Beanstalk failed to install Python package using requirements.txt - Firebase-Admin

Question

I have searched similar questions, but no answers solve my problem. I am trying to install firebase-admin using pip. Everything works well locally, but when I push to aws elastic beanstalk it gives me the error below When I remove firebase-admin and its dependencies everything works fine. I think it may specifically have an error with grpcio being installed but not completely sure, and firebase needs it installed to work. Some answers say that firebase-admin is dynamic and aws else does not support that. I am wondering if there is a solution to this problem so I can install firebase. Thank you

2020-06-21 04:13:35    INFO    Environment update is starting.      
2020-06-21 04:13:40    INFO    Deploying new version to instance(s).
2020-06-21 04:13:51    ERROR   Your requirements.txt is invalid. Snapshot your logs for details.
2020-06-21 04:13:55    ERROR   [Instance: i-06c2884999a9d6064] Command failed on instance. Return code: 1 Output: (TRUNCATED)...)
File "/usr/lib64/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt' returned non-zero exit status 1. 
Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2020-06-21 04:13:55    INFO    Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].

My requirments.txt file:

APScheduler==3.6.3
asgiref==3.2.10
braintree==4.1.0
CacheControl==0.12.6
cachetools==4.1.0
certifi==2020.4.5.2
chardet==3.0.4
defusedxml==0.6.0
Django==3.0.7
django-allauth==0.42.0
django-cors-headers==3.4.0
django-mysql==3.7.0
django-rest-auth==0.9.5
djangorestframework==3.11.0
firebase-admin==4.3.0
google-api-core==1.21.0
google-api-python-client==1.9.3
google-auth==1.18.0
google-auth-httplib2==0.0.3
google-cloud-core==1.3.0
google-cloud-firestore==1.7.0
google-cloud-storage==1.29.0
google-resumable-media==0.5.1
googleapis-common-protos==1.52.0
grpcio==1.29.0
httplib2==0.18.1
idna==2.9
msgpack==1.0.0
mysqlclient==1.4.6
numpy==1.18.5
oauthlib==3.1.0
protobuf==3.12.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyJWT==1.7.1
python3-openid==3.1.0
pytz==2020.1
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
six==1.15.0
sqlparse==0.3.1
twilio==6.42.0
tzlocal==2.1
uritemplate==3.0.1
urllib3==1.25.9

Some of the Log File:

     gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DOPENSSL_NO_ASM=1 -D_WIN32_WINNT=1536 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_linux -Ithird_party/boringssl-with-bazel/src/include -Ithird_party/upb -Isrc/core/ext/upb-generated -Ithird_party/zlib -I/usr/include/python3.6m -c src/python/grpcio/grpc/_cython/cygrpc.cpp -o python_build/temp.linux-x86_64-3.6/src/python/grpcio/grpc/_cython/cygrpc.o -std=c++11 -std=gnu99 -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread
  gcc: error trying to exec 'cc1plus': execvp: No such file or directory
  creating tmp/tmpi4o4pn51
  gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m -c /tmp/tmpi4o4pn51/a.c -o tmp/tmpi4o4pn51/a.o
  Traceback (most recent call last):
    File "/usr/lib64/python3.6/distutils/unixccompiler.py", line 127, in _compile
      extra_postargs)
    File "/tmp/pip-build-29tk6f4m/grpcio/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
      _classic_spawn(self, command)
    File "/usr/lib64/python3.6/distutils/ccompiler.py", line 909, in spawn
      spawn(cmd, dry_run=self.dry_run)
    File "/usr/lib64/python3.6/distutils/spawn.py", line 36, in spawn
      _spawn_posix(cmd, search_path, dry_run=dry_run)
    File "/usr/lib64/python3.6/distutils/spawn.py", line 159, in _spawn_posix
      % (cmd, exit_status))
  distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/tmp/pip-build-29tk6f4m/grpcio/src/python/grpcio/commands.py", line 262, in build_extensions
      build_ext.build_ext.build_extensions(self)
    File "/usr/lib64/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
      self._build_extensions_serial()
    File "/usr/lib64/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
      self.build_extension(ext)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
      _build_ext.build_extension(self, ext)
    File "/usr/lib64/python3.6/distutils/command/build_ext.py", line 533, in build_extension
      depends=ext.depends)
    File "/usr/lib64/python3.6/distutils/ccompiler.py", line 574, in compile
      self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
    File "/tmp/pip-build-29tk6f4m/grpcio/src/python/grpcio/commands.py", line 246, in new_compile
      pp_opts)
    File "/usr/lib64/python3.6/distutils/unixccompiler.py", line 129, in _compile
      raise CompileError(msg)
  distutils.errors.CompileError: command 'gcc' failed with exit status 1
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-29tk6f4m/grpcio/setup.py", line 400, in <module>
      cmdclass=COMMAND_CLASS,
    File "/usr/lib64/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
      return orig.install.run(self)
    File "/usr/lib64/python3.6/distutils/command/install.py", line 593, in run
      self.run_command('build')
    File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/lib64/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 75, in run
      _build_ext.run(self)
    File "/usr/lib64/python3.6/distutils/command/build_ext.py", line 339, in run
      self.build_extensions()
    File "/tmp/pip-build-29tk6f4m/grpcio/src/python/grpcio/commands.py", line 267, in build_extensions
      "Failed `build_ext` step:\n{}".format(formatted_exception))
  commands.CommandError: Failed `build_ext` step:
  Traceback (most recent call last):
    File "/usr/lib64/python3.6/distutils/unixccompiler.py", line 127, in _compile
      extra_postargs)
    File "/tmp/pip-build-29tk6f4m/grpcio/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
      _classic_spawn(self, command)
    File "/usr/lib64/python3.6/distutils/ccompiler.py", line 909, in spawn
      spawn(cmd, dry_run=self.dry_run)
    File "/usr/lib64/python3.6/distutils/spawn.py", line 36, in spawn
      _spawn_posix(cmd, search_path, dry_run=dry_run)
    File "/usr/lib64/python3.6/distutils/spawn.py", line 159, in _spawn_posix
      % (cmd, exit_status))
  distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/tmp/pip-build-29tk6f4m/grpcio/src/python/grpcio/commands.py", line 262, in build_extensions
      build_ext.build_ext.build_extensions(self)
    File "/usr/lib64/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
      self._build_extensions_serial()
    File "/usr/lib64/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
      self.build_extension(ext)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
      _build_ext.build_extension(self, ext)
    File "/usr/lib64/python3.6/distutils/command/build_ext.py", line 533, in build_extension
      depends=ext.depends)
    File "/usr/lib64/python3.6/distutils/ccompiler.py", line 574, in compile
      self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
    File "/tmp/pip-build-29tk6f4m/grpcio/src/python/grpcio/commands.py", line 246, in new_compile
      pp_opts)
    File "/usr/lib64/python3.6/distutils/unixccompiler.py", line 129, in _compile
      raise CompileError(msg)
  distutils.errors.CompileError: command 'gcc' failed with exit status 1
    Command "/opt/python/run/venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-29tk6f4m/grpcio/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-ox9mho6_-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/python/run/venv/include/site/python3.6/grpcio" failed with error code 1 in /tmp/pip-build-29tk6f4m/grpcio/
You are using pip version 9.0.1, however version 20.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
2020-06-19 20:06:16,263 ERROR    Error installing dependencies: Command '/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt' returned non-zero exit status 1
Traceback (most recent call last):
File "/opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py", line 22, in main
  install_dependencies()
 File "/opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py", line 18, in install_dependencies
  check_call('%s install -r %s' % (os.path.join(APP_VIRTUAL_ENV, 'bin', 'pip'), requirements_file), shell=True)
 File "/usr/lib64/python2.7/subprocess.py", line 190, in check_call
  raise CalledProcessError(retcode, cmd)
 CalledProcessError: Command '/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt' returned non-zero exit status 1 (Executor::NonZeroExitStatus)

[2020-06-19T20:06:16.273Z] INFO  [6647]  - [Application update app-200619_130554@310/AppDeployStage0/AppDeployPreHook/03deploy.py] : Activity failed.
[2020-06-19T20:06:16.273Z] INFO  [6647]  - [Application update app-200619_130554@310/AppDeployStage0/AppDeployPreHook] : Activity failed.
[2020-06-19T20:06:16.273Z] INFO  [6647]  - [Application update app-200619_130554@310/AppDeployStage0] : Activity failed.
[2020-06-19T20:06:16.275Z] INFO  [6647]  - [Application update app-200619_130554@310] : Completed activity. Result:
Application update - Command CMD-AppDeploy failed
[2020-06-19T20:07:34.137Z] INFO  [6815]  - [CMD-BundleLogs] : Starting activity...
[2020-06-19T20:07:34.138Z] INFO  [6815]  - [CMD-BundleLogs/AddonsBefore] : Starting activity...
[2020-06-19T20:07:34.138Z] INFO  [6815]  - [CMD-BundleLogs/AddonsBefore] : Completed activity.
[2020-06-19T20:07:34.138Z] INFO  [6815]  - [CMD-BundleLogs/BundleLogs] : Starting activity...
[2020-06-19T20:07:34.138Z] INFO  [6815]  - [CMD-BundleLogs/BundleLogs/BundleLogs] : Starting activity...

Answer 1

pip install grpcio==1.26.0rc1

The solution is to downgrade grpcio. Unfortunately, since Django 2.1 is the latest compatible version with Elastic Beanstalk Python 3.6 platform, we sometimes have to downgrade other packages accordingly.

Hopefully AWS upgrades the platform for use with the latest Django eventually...

Answer 2

I wasn't using firebase-admin , but I am trying to deploy a Django application via Elastic Beanstalk and also ran into this error during the step when requirements.txt was being installed.

To add to Connor's answer, I moved grpcio==1.26.0rc1 up in my requirements.txt file until it was above the package which must have been installing an incompatible version of grpcio :

backcall==0.1.0
certifi==2019.3.9
chardet==3.0.4
decorator==4.3.0
dj-database-url==0.5.0
Django==2.2
django-cors-headers==2.4.0
djangorestframework==3.9.1
djangorestframework-jwt==1.11.0
grpcio==1.26.0rc1

FWIW it now works fine with Django 2.2 on the latest AMI (Python 3.6 running on 64bit Amazon Linux/2.9.16).