How can I make use of swap space/virtual RAM in Jupyter lab/notebook?

Question

I am running processes in Jupyter (lab) in a JupyterHub-created container running on Kubernetes.

The processes are too RAM-intensive, to the extent that the pod sometimes gets evicted due to an OOM.

Without modifications to my code/algorithms etc., how can I in the general case tell Jupyter(Lab) to use swap space/virtual memory when a predefined RAM limit is reached?

PS This question has no answer mentioning swap space - Jupyter Lab freezes the computer when out of RAM - how to prevent it?

Answer 1

You can't actively control swap space.

In Kubernetes specifically, you just don't supply a memory limit for the Kubernetes pod. That would at least not kill it because of OOM (out of memory). However, I doubt it would work because this will make the whole node go out of RAM, then swap and become extremely slow and thus at some point declared dead by the Kubernetes master. Which in turn will cause the Pod to run somewhere else and start all over again.

A more scalable approach for you might be to use out-of-core algorithms, that can operate on disk directly (so just attach a PV/PVC to your pod), but that depends on the algorithm or process you're using.