Run ckan using Dockerfile from Docker Hub

Question

Trying to install ckan from Dockerfile in:

Docker Community Edition Version 17.06.2-ce-mac27 (19124) Channel: stable 428bd6ceae*

FIRST ATTEMPT

These are the steps followed:

$ docker pull ckan/solr
$ docker pull ckan/ckan
$ docker pull ckan/postgresql

After downloaded the images I get:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ckan/solr           latest              4acd7db7b1f7        3 days ago          517MB
ckan/ckan           latest              77dd30c92740        3 days ago          642MB
ckan/postgresql     latest              3c3ecd94ae7e        3 days ago          265MB

For each Dockerfile I made:

For solr

$ docker build .  (for SOLR)
Step 7/9
ADD failed: stat /var/lib/docker/tmp/docker-builder234804113/solrconfig.xml: no such file or directory

For postgresql

$ docker build .  (for POSTGRESQL)
Successfully built 8b296a6e3153

For ckan

$ docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan

Error in Step 7/26 :

Get:1 http://security.debian.org/ jessie/updates/main perl-base amd64 5.20.2-3+deb8u9 [1226 kB]
debconf: delaying package configuration, since apt-utils is not installed

And Fail in:

Step 12/26 : ADD ./requirements.txt $CKAN_HOME/src/ckan/requirements.txt
ADD failed: stat /var/lib/docker/tmp/docker-builder814560705/requirements.txt: no such file or directory

SECOND ATTEMPT

Removed images

Pull again images

$ docker pull ckan/solr
$ docker pull ckan/ckan
$ docker pull ckan/postgresql

And Launch:

$ docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan

for the Dockerfile in: https://hub.docker.com/r/ckan/ckan/~/dockerfile/

and I get the same error in the same step:

Step 12/26 : ADD ./requirements.txt $CKAN_HOME/src/ckan/requirements.txt
ADD failed: stat /var/lib/docker/tmp/docker-builder254648764/requirements.txt: no such file or directory

but the log in Docker Hub is correct.

Am I doing something wrong?????

Dockerfile

FROM debian:jessie
MAINTAINER Open Knowledge

ENV CKAN_HOME /usr/lib/ckan/default
ENV CKAN_CONFIG /etc/ckan/default
ENV CKAN_STORAGE_PATH /var/lib/ckan
ENV CKAN_SITE_URL http://localhost:5000

# Install required packages
RUN apt-get -q -y update && apt-get -q -y upgrade && DEBIAN_FRONTEND=noninteractive apt-get -q -y install \
        python-dev \
        python-pip \
        python-virtualenv \
        libpq-dev \
        git-core \
        build-essential \
        libssl-dev \
        libffi-dev \
    && apt-get -q clean

# SetUp Virtual Environment CKAN
RUN mkdir -p $CKAN_HOME $CKAN_CONFIG $CKAN_STORAGE_PATH
RUN virtualenv $CKAN_HOME
RUN ln -s $CKAN_HOME/bin/pip /usr/local/bin/ckan-pip
RUN ln -s $CKAN_HOME/bin/paster /usr/local/bin/ckan-paster

# SetUp Requirements
ADD ./requirements.txt $CKAN_HOME/src/ckan/requirements.txt
RUN ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/requirements.txt

# TMP-BUGFIX https://github.com/ckan/ckan/issues/3388
ADD ./dev-requirements.txt $CKAN_HOME/src/ckan/dev-requirements.txt
RUN ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/dev-requirements.txt

# TMP-BUGFIX https://github.com/ckan/ckan/issues/3594
RUN ckan-pip install --upgrade urllib3

# SetUp CKAN
ADD . $CKAN_HOME/src/ckan/
RUN ckan-pip install -e $CKAN_HOME/src/ckan/
RUN ln -s $CKAN_HOME/src/ckan/ckan/config/who.ini $CKAN_CONFIG/who.ini

# SetUp EntryPoint
COPY ./contrib/docker/ckan-entrypoint.sh /
RUN chmod +x /ckan-entrypoint.sh
ENTRYPOINT ["/ckan-entrypoint.sh"]

# Volumes
VOLUME ["/etc/ckan/default"]
VOLUME ["/var/lib/ckan"]
EXPOSE 5000
CMD ["ckan-paster","serve","/etc/ckan/default/ckan.ini"]

UPDATE 1

As suggested, added the content of the ckan package from: http://packaging.ckan.org/

Copied to my Dockerfile directory /data/usr/lib/ckan/default/src/ckan/*

New error in step 15 (see comments)

UPDATE 2

Cloned ckan from: https://github.com/ckan/ckan

$ docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan

Result:

Step 26/26 : CMD ckan-paster serve /etc/ckan/default/ckan.ini
 ---> Running in eeeb6ace6ee1
 ---> 3cd87cf4a1af
Removing intermediate container eeeb6ace6ee1
Successfully built 3cd87cf4a1af
Successfully tagged ckan:latest
docker: Error response from daemon: could not get container for db: No such container: db.

Trying to access to ckan:

docker run -it ckan bash

I get this error:

Distribution already installed:
  ckan 2.8.0a0 from /usr/lib/ckan/default/src/ckan
Creating /etc/ckan/default/ckan.ini
Now you should edit the config files
  /etc/ckan/default/ckan.ini
Traceback (most recent call last):
  File "/usr/local/bin/ckan-paster", line 11, in <module>
    sys.exit(run())
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 348, in command
    self._load_config(cmd!='upgrade')
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 321, in _load_config
    self.site_user = load_config(self.options.config, load_site_user)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 230, in load_config
    load_environment(conf.global_conf, conf.local_conf)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 111, in load_environment
    p.load_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 129, in load_all
    unload_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 182, in unload_all
    unload(*reversed(_PLUGINS))
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 210, in unload
    plugins_update()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 121, in plugins_update
    environment.update_config()
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 289, in update_config
    engine = sqlalchemy.engine_from_config(config, client_encoding='utf8')
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 428, in engine_from_config
    return create_engine(url, **options)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 50, in create
    u = url.make_url(name_or_url)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 194, in make_url
    return _parse_rfc1738_args(name_or_url)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 240, in _parse_rfc1738_args
    return URL(name, **components)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 60, in __init__
    self.port = int(port)
ValueError: invalid literal for int() with base 10: ''

UPDATE 3 (matt fullerton suggestion)

I have cloned https://github.com/parksandwildlife/ckan.git in a folder called ckan

I have cloned https://github.com/parksandwildlife/ckan/tree/3649-docker-upgrade in another folder called ckan-3649

In ckan folder

docker build . -t ckan && docker run -d -p 80:5000 --link db:db --link redis:redis --link solr:solr ckan

In Step 15/26 :

 RUN ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/dev-requirements.txt

ERROR:

Cleaning up...
Command /usr/lib/ckan/default/bin/python2 -c "import setuptools, tokenize;__file__='/tmp/pip-build-BLM6DJ/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-Vynm83-record/install-record.txt --single-version-externally-managed --compile --install-headers /usr/lib/ckan/default/include/site/python2.7 failed with error code 1 in /tmp/pip-build-BLM6DJ/cryptography
Storing debug log for failure in /root/.pip/pip.log
The command '/bin/sh -c ckan-pip install --upgrade -r $CKAN_HOME/src/ckan/dev-requirements.txt' returned a non-zero code: 1

In ckan-3649/contrib/docker "docker-compose up"

$ docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                    NAMES
b6d7a5a595ca        redis:latest                "docker-entrypoint..."   9 minutes ago       Up 9 minutes        6379/tcp                 redis
4551ccea28f4        ckan/solr:latest            "docker-entrypoint..."   9 minutes ago       Up 9 minutes        8983/tcp                 solr
86cea84b6ab5        docker_db                   "docker-entrypoint..."   9 minutes ago       Up 9 minutes        5432/tcp                 db
fca64f1bee5a        clementmouchet/datapusher   "python datapusher..."   9 minutes ago       Up 9 minutes        0.0.0.0:8800->8800/tcp   datapusher

But I think ckan image is missing....

Answer 1

There is a major upgrade of the docker functionality in CKAN underway. I would suggest cloning CKAN from here: https://github.com/parksandwildlife/ckan

git clone https://github.com/parksandwildlife/ckan.git

Then checkout the origin/3649-docker-upgrade branch ( https://github.com/parksandwildlife/ckan/tree/3649-docker-upgrade )

git checkout origin/3649-docker-upgrade

Then use docker-compose in the contrib/docker folder:

docker-compose up

This should assemble Solr, Postgres etc. for you.

Comments on mileage with this at https://github.com/ckan/ckan/pull/3692 will also be appreciated.