Errors trying to install Acumos Boreas release

Question

Having a hope that the Acumos Boreas OneClick (and other install scripts) actually work now...

I am trying, on behalf of the AI4EU project (Task 3.2) to install the Acumos Boreas release on an Ubuntu 18.04 server again. Unfortunately my hope is dwindling...

I am following the procedure under section 2.1.2 here: https://docs.acumos.org/en/boreas/submodules/system-integration/docs/oneclick-deploy/user-guide.html#host-vm-preparation

I am starting from a fresh Ubuntu 18.04 Virtual Machine (created with 32G Memory, 12 Cores and 300 GB disk).

Doing this (and typing the sudo password when prompted):

git clone https://gerrit.acumos.org/r/system-integration
cd system-integration/tools/
bash setup_docker.sh
if [[ "$(id -nG "$USER" | grep docker)" == "" ]]; then sudo usermod -aG docker $USER; fi
# Logged out and in again and verified that my user is in the docker group
cd system-integration/tools/
bash setup_k8s_stack.sh setup
cd
bash system-integration/AIO/setup_prereqs.sh k8s acumos.tele.no $USER generic 2>&1 | tee aio_prep.log
# When "Prerequisites setup is complete" messages is displayed I continue with
cd system-integration/AIO
bash oneclick_deploy.sh 2>&1 | tee aio_deploy.log

The deployment fails with the following error message:

....
oneclick_deploy.sh setup_federation:233 (Tue Aug 20 13:47:04 UTC 2019) CDS API is not yet ready; waiting 10 seconds
+ t=300
+ sleep 10
++ curl -k -u ccds_client:27f928e9-cdde-4483-b3c9-7da074972908 https://acumos.tele.no/ccds/peer
++ grep -c numberOfElements
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   203  100   203    0     0   7000      0 --:--:-- --:--:-- --:--:--  7000
+ [[ 0 -eq 0 ]]
+ [[ 300 -eq 300 ]]
+ fail 'CDS API is not ready after 300 seconds'
+ set +x

When I use the Kubernetes Dashboard to see what actually fails I do see that the common-dataservice pod fails.

I can also see that the docker_proxy apparently crashes regularly as well as some other pods.

All install and error logs are available here: https://www.dropbox.com/sh/61snwd26zbixwl3/AAAWcfBKnIwNkRghXSMQayrEa?dl=0

If anyone is able to guide me on how to make an install of Acumos Boreas for the AI4EU project (WP3) to explore on it would be highly appreciated.

Answer 1

Arne, make sure that you can connect to the host 'acumos' from within the running CDS container. If this is a cloud service VM, you may have to open a security group rule to allow the VM to connect to itself on its public IP (sounds implicit, but isn't always the default). You can test this out via

kubectl exec -it -n acumos $(kubectl get pods -n acumos -l app=sv-scanning | awk '/sv-scanning/{print $1}') -- curl http://acumos:30001

NOTE: This command example uses the sv-scanning container because it has curl installed... the test is however to see if any container can connect to MariaDB. By default, if the Acumos domain is not DNS-resolvable, as indicated by an entry in the /etc/hosts file, a hostAlias will be added to each deployment template.

(update) I added a script to create a debug container (ubuntu based) and run it as a pod under the namespace. That way you can add any tools you want to debug, eg I have already installed curl, j1, netcat. Use that container to verify connectivity to http://acumos:30001 , check out the CDS logs, etc.

#!/bin/bash
kubectl delete deployment -n acumos debug
while [[ "$(kubectl get pods -n acumos -l app=debug)" != "" ]] ; do
  echo 'waiting...'; sleep 10
done
cat <<EOF >debug.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: acumos
  name: debug
spec:
  selector:
    matchLabels:
      app: debug
  replicas: 1
  template:
    metadata:
      labels:
        app: debug
    spec:
      containers:
      - name: debug
        image: ubuntu
        command: ['/bin/bash', '-c']
        args:
        - apt-get update; apt-get install -y curl jq netcat;
          sleep 3600;
        volumeMounts:
        - mountPath: /logs
          name: logs
      restartPolicy: Always
      volumes:
      - name: logs
        persistentVolumeClaim:
         claimName: logs
EOF
kubectl create -f debug.yaml
# Wait till running
while [[ $(kubectl get pods -n acumos -o yaml -l app=debug | grep -c 'phase: Running') -eq 0 ]]; do
  echo 'waiting...'; sleep 10
done
kubectl exec -it -n acumos $(kubectl get pods -n acumos -l app=debug -o name | sed 's/pod\///') -- bash