python code order of magnitude faster than compiled C code for simple import statement

Question

I'm baffled by the performance difference of the following two semantically equivalent(?) programs.

Python:

#test.py
import time
starttime=time.time()
import tensorflow 
print(f"Import took: {time.time() - starttime}")

C using CPython

//test.c
#include <Python.h>
#include <stdio.h> 
#include <time.h> 
int main(int argc, char *argv[])
{
    Py_Initialize();
    clock_t t = clock();
    PyObject* pModule = PyImport_ImportModule("tensorflow");
    double time_taken = ((double)clock() - t)/CLOCKS_PER_SEC;
    printf("Import took:  %f\n", time_taken);
    return 0;
}

Now compare the two:

cl.exe /I"C:\Program Files\Python37\include" test.c
link test.obj python37.lib /LIBPATH:"C:\Program Files\Python37\libs"
.\test.exe

2020-04-17 13:00:51.598550: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found 2020-04-17 13:00:51.606296: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

Import took: 23.160523891448975

python test.py

2020-04-17 13:01:19.525648: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found 2020-04-17 13:01:19.530726: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

Import took: 2.3172824382781982

Not that it should matter much but my tensorflow version is 1.15 Why is the python VM code so much faster than the compiled CPython code? Does the python vm add some magic that PyImport_ImportModule doesn't?

Answer 1

PyImport_ImportModule is part of the C ABI, which means it can have performance penalty: https://docs.python.org/3/c-api/stable.html#stable-abi-list