erlang error when erlang spawning python instance, with erlport

Question

I try to figure out how many python instance/processe I can create from erlang.

I manage to create 500 instance. when all processe are created the cpu is near to 0%.

1 python instance take 6.2MB of ram. 400 python instance take 2.4 GB of ram. 500 python instance take 3.1 GB of ram.

when i try to create 600 python instance i got this Error:

/usr/lib/erlang/erts-7.0/bin/child_setup: error while loading shared libraries: libc.so.6: cannot open shared object file: Error 24 ** exception error: no match of right hand side value {error, {invalid_python,"/usr/bin/python"}} in function spawn_python:spawn_python/2 (spawn_python.erl, line 10)

theoretically i need 6.2MB*600 = 3.7 GB of free ram (i got them).

Here is the code i use. it need erlport : http://erlport.org/downloads/

%% how i test it

spawn_python:spawn_python(100). spawn_python:spawn_python(100). spawn_python:spawn_python(100). spawn_python:spawn_python(100). spawn_python:spawn_python(100). spawn_python:spawn_python(100). ERROR here

%%%%%%%% spawn_python.erl  %%%%%%%%%%%%
-module(spawn_python).
-export([spawn_python/1, spawn_python/2]).

%% spawn N python process
spawn_python(N) -> spawn_python(N, []).

spawn_python(0, PyProcPids) ->
    PyProcPids;
spawn_python(N, PyProcPids) ->
    {ok, PyPid} = python:start(),
    PyProcPid = spawn(fun() -> python_loop(PyPid) end),
    io:format("Started Erlang/Python process -- PyProcPid: ~p~n", [PyProcPid]),
    spawn_python(N - 1, [PyProcPid | PyProcPids]).

python_loop(PyPid) ->
    receive
        {stop, From} ->
            python:stop(PyPid),
            From ! ok
    end.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Answer 1

Each spawned port needs some pipes to communicate with. Each pipe uses file descriptor (two of them but one is closed soon) and there is a limit how much file descriptors you can have opened. The number of maximum opened file descriptors is set by underlining OS (see ulimit syscall defined in POSIX.1-2001 or ulimit command of shell for unix like systems).