Python Script Stops Silently

Question

I'm running Python scripts as child processes, spawned using Nodejs.

When running locally, or locally using Docker / Kube.netes installation, it works as expected and completes all functions in the script. When running the container in Kube.netes Azure, the script silently stops / fails at just under 1 hour, without any exceptions or errors logged.

Memory & CPU usage stays below 30% max, container as a whole doesn't fail. When running ps -fA | grep python ps -fA | grep python I can see the script running after it has been spawned. Script doesn't show anymore after it fails / stops silently. The 'exit' and 'close' events within Nodejs for the spawned processes do not fire.

Any advice on how to troubleshoot would be much appreciated.

EDIT: Nodejs spawn

import {/* inject, */ BindingScope, injectable} from '@loopback/core';

const path = require('path');

const spawn = require('child_process').spawn;

@injectable({scope: BindingScope.TRANSIENT})
export class PythonService {
  constructor() {} 
  stopPython(valuationId) {}

  executePython(id: string) {
    const filepath = path.resolve(process.env.PY_PATH);

    const ls = spawn('python', [filepath, id]);

    ls.stdout.on('data', function (data) {
      console.log('stdout: ' + data.toString());
    });

    ls.stderr.on('data', function (data) {
      console.log('stderr: ' + data.toString());
    });

    ls.error.on('error', function (data) {
      console.log('error: ' + data.toString());
    });

    ls.on('exit', function (code) {
      console.log('child process exited with code ' + code.toString());
    });

    ls.on('close', code => {
      console.log(`child process exited with code ${code}`);
    });
  }
}

EDIT: Dockerfile

# Pull base image
FROM python:3.7-slim

# Set installation environment variables
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV NODE_VERSION=12.20.0

# Install NVM for later use to install Node and NPM
RUN apt-get update && apt install -y curl
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
ENV NVM_DIR=/root/.nvm
RUN . "$NVM_DIR/nvm.sh" && nvm install ${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm use v${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm alias default v${NODE_VERSION}
ENV PATH="/root/.nvm/versions/node/v${NODE_VERSION}/bin/:${PATH}"

# Create app directory (with user `node`)
RUN mkdir -p /home/node/app

# Set work directory
WORKDIR /home/node/app

# Install python dependencies
COPY  requirements.txt /home/node/app/
RUN pip install -r requirements.txt
RUN pip install swifter

# Install node app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY  package*.json ./
RUN npm install

# Bundle app source code
COPY . .

# Build node app
RUN  npm run build

# Expose ports
EXPOSE ${DB_PORT}
EXPOSE ${API_PORT}
EXPOSE ${SOCKET_PORT}

CMD [ "node", "." ]

Python v 3.7.11 Nodejs v 12.20

Answer 1

Unix was killing the Python processes due to high memory usage, I was able to find OOM errors in the system logs by using ssh into the pod, then using dmesg for the kill logs and ps aux --sort -pmem to see the memory usage in the pod.

Reason for OOM was that the default memory allocated to Nodejs was considerably higher than the normal 2GB limit, which decreased the available memory for Python. Decreasing Nodejs memory allocation or removing exlusive Nodejs memory allocation solves the issue.