Reference count of function arguments in CPython 3.6

Question

In order to find out if a parameter passed to a function is a "temporary" (only passed into the function) or also referenced outside I use Py_REFCNT . This is done in a C extension package, but for easier reproducibility I decided to provide a Cython implementation based on IPython magic here.

It seems like something changed for functions that accept multiple arguments (it still works as expected for functions that only take one argument) between CPython 3.5 and CPython 3.6:

In [1]: %load_ext cython

In [2]: %%cython
   ...: cdef extern from "Python.h":
   ...:     Py_ssize_t Py_REFCNT(object o)
   ...:
   ...: cpdef func(o, p):
   ...:     return Py_REFCNT(o)

When I run the code on 3.5 it gives me, the expected result:

>>> import numpy as np
>>> func(np.ones(3), np.ones(3))
1

But with 3.6 it gives me 2 :

>>> import numpy as np
>>> func(np.ones(3), np.ones(3))
2

---

In the comments I was asked about the C code so here it is:

static PyObject *
GetRefCount(PyObject *m, PyObject *args) {
    if (PyTuple_CheckExact(args) && PyTuple_Size(args) > 0) {
        Py_ssize_t reference_count = Py_REFCNT(PyTuple_GET_ITEM(args, 0));
        return PyLong_FromSsize_t(reference_count);
    }
    PyErr_SetString(PyExc_TypeError, "wrong input");
    return NULL;
}

And the method definition:

    {"getrefcount",                                     /* ml_name */
     (PyCFunction)GetRefCount,                          /* ml_meth */
     METH_VARARGS,                                      /* ml_flags */
     ""                                                 /* ml_doc */
     },

The results are the same:

>>> import numpy as np
>>> getrefcount(np.ones(3))  # 3.5
1
>>> getrefcount(np.ones(3))  # 3.6
2

I would like to know where (and why) the reference count is incremented in 3.6. I have looked through the CPython source code / the Python issue tracker but I couldn't find an answer.

Answer 1

On Python 3.5, the arguments happen to be cleared from the caller's stack by the time your function is executed. On Python 3.6, the arguments happen to still be on the caller's stack as well as in your function's argument tuple.

On Python 3.5, your function call goes through here :

    else {
        PyObject *callargs;
        callargs = load_args(pp_stack, na);
        if (callargs != NULL) {
            READ_TIMESTAMP(*pintr0);
            C_TRACE(x, PyCFunction_Call(func,callargs,NULL));
            READ_TIMESTAMP(*pintr1);
            Py_XDECREF(callargs);
        }
        else {
            x = NULL;
        }
    }

which removes the arguments from the stack to build the argument tuple:

static PyObject *
load_args(PyObject ***pp_stack, int na)
{
    PyObject *args = PyTuple_New(na);
    PyObject *w;

    if (args == NULL)
        return NULL;
    while (--na >= 0) {
        w = EXT_POP(*pp_stack);
        PyTuple_SET_ITEM(args, na, w);
    }
    return args;
}

On 3.6, your function call goes through here :

if (PyCFunction_Check(func)) {
    PyThreadState *tstate = PyThreadState_GET();

    PCALL(PCALL_CFUNCTION);

    stack = (*pp_stack) - nargs - nkwargs;
    C_TRACE(x, _PyCFunction_FastCallKeywords(func, stack, nargs, kwnames));
}

which goes through here

PyObject *
_PyCFunction_FastCallKeywords(PyObject *func, PyObject **stack,
                              Py_ssize_t nargs, PyObject *kwnames)
{
    ...

    result = _PyCFunction_FastCallDict(func, stack, nargs, kwdict);
    Py_XDECREF(kwdict);
    return result;
}

which goes through here :

case METH_VARARGS:
case METH_VARARGS | METH_KEYWORDS:
{
    /* Slow-path: create a temporary tuple */
    ...

    tuple = _PyStack_AsTuple(args, nargs);

    ...
}

which goes through here :

for (i=0; i < nargs; i++) {
    PyObject *item = stack[i];
    Py_INCREF(item);
    PyTuple_SET_ITEM(args, i, item);
}

which leaves the arguments on the stack and builds a tuple with new references to the arguments.