如何在 Linux 服务器上使用 docker-compose 安装 phantomjs 和 selenium？

Question

I am using selenium and phantomjs for my web scraper. All works great with my test windows app. Trying to add this code update to my main app, deployed with docker-compose, and i get this: selenium.common.exceptions.WebDriverException: Message: 'phantomjs' executable needs to be in PATH.

How should i fix this? currently my docker-compose.yml has this code:

version: '3.1'

services:

  tgbot:
    container_name: bot
    build:
      context: .
    command: python app.py
    restart: always
    environment:
      WEBAPP_PORT: 3001
    env_file:
      - ".env"
    # bot start after load db
    ports:
      - 8443:3001
    networks:
      - botnet

  phantomjs:
    image: shufo/phantomjs
    command: --webdriver 8901

networks:
      botnet:
        driver: bridge

And my python code:

from selenium import webdriver
driver = webdriver.PhantomJS()

Dockefile:

FROM python:latest

RUN mkdir /src
WORKDIR /src
COPY requirements.txt /src
RUN pip install -r requirements.txt
COPY . /src

PS i am using phantomjs because webpage i am scraping has JS. doesnt work with chrome

Answer 1

There are few problems with your configuration:

1. Your bot code is working in different container. Not in that one that launches phantomjs. This is why it cannot find the executable.
2. You run phantomjs container not within the same network as your code
3. There are useless configs which seem to be copypasted from some other example.
4. You force your containers to restart. It will be restarting even after the successful exit code.

So here is the complete example how to run everything:

1. Create empty folder myfolder and put there app.py with the following content:

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
driver = webdriver.Remote(command_executor='http://phantomjs:8901/wd/hub/',desired_capabilities=DesiredCapabilities.PHANTOMJS)

2. Put requirements.txt file to myfolder
3. Put following Dockerfile to myfolder :

FROM python:latest

WORKDIR /configs
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

4. Put following docker-compose.yml to myfolder :

version: '3.1'

services:

  tgbot:
    build: .
    container_name: bot
    volumes:
      - .:/apps
    command: python /apps/app.py
    depends_on:
      - phantomjs
    networks:
      - botnet

  phantomjs:
    container_name: phantomjs
    image: shufo/phantomjs
    command: --webdriver 8901
    networks:
      - botnet

networks:
  botnet:
    driver: bridge

4. cd myfloder , docker-compose up

Output:

phantomjs    | [INFO  - 2020-11-10T15:18:11.049Z] GhostDriver - Main - running on port 8901
phantomjs    | [INFO  - 2020-11-10T15:18:11.425Z] Session [f2091fe0-2367-11eb-bcd7-956b9cd40e54] - page.settings - {"XSSAuditingEnabled":false,"javascriptCanCloseWindows":true,"javascriptCanOpenWindows":true,"javascriptEnabled":true,"loadImages":true,"localToRemoteUrlAccessEnabled":false,"userAgent":"Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1","webSecurityEnabled":true}
phantomjs    | [INFO  - 2020-11-10T15:18:11.425Z] Session [f2091fe0-2367-11eb-bcd7-956b9cd40e54] - page.customHeaders:  - {}
phantomjs    | [INFO  - 2020-11-10T15:18:11.425Z] Session [f2091fe0-2367-11eb-bcd7-956b9cd40e54] - Session.negotiatedCapabilities - {"browserName":"phantomjs","version":"2.1.1","driverName":"ghostdriver","driverVersion":"1.2.0","platform":"linux-unknown-64bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
phantomjs    | [INFO  - 2020-11-10T15:18:11.425Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: f2091fe0-2367-11eb-bcd7-956b9cd40e54
bot exited with code 0