了解python引用计数以调试c-扩展

Question

I am writing a c-extension and want to test it in pytest.

Part of what I am testing is whether the reference counts are correct on my objects. Thus I build a small test in pure python that puzzles me...

from Ipython i get:

In [1]: x = 153513514215

In [2]: import sys

In [3]: sys.getrefcount(x)
Out[3]: 2

So fare so good, 1 reference from assignment and 1 from the caller.

However the following script (stackoverflow_test.py) gives the following results

import sys

def test_ref_count_int():
    x = 677461248192962146784178
    assert sys.getrefcount(x) == 2

def test_ref_count_str():
    y = 'very long and probbably very unique string'
    assert sys.getrefcount(y) == 2

def normal_te_st():
    x = 222677461248192962146784178
    y = '!!!!very long and probbably unique string!!!!'
    print ('x refcount = {}'.format(sys.getrefcount(x)))
    print ('y refcount = {}'.format(sys.getrefcount(y)))

if __name__ == '__main__':
    normal_te_st()

when I run it as a normal python script

$ python3 stackoverflow_test.py
x refcount = 4
y refcount = 4

Why 4 and not 2?.

When I run it with pytest

$ python3 -m pytest stackoverflow_test.py
=================== test session starts ===================
platform linux -- Python 3.4.3, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /opt/projects/0001_Intomics/00005_TextMining/jcr/textmining/tests, inifile:
collected 2 items

stackoverflow_test.py FF

======================== FAILURES =========================
___________________ test_ref_count_int ____________________

    def test_ref_count_int():
        x = 677461248192962146784178
>       assert sys.getrefcount(x) == 2
E       assert 3 == 2
E        +  where 3 = <built-in function getrefcount>(677461248192962146784178)
E        +    where <built-in function getrefcount> = sys.getrefcount

stackoverflow_test.py:7: AssertionError
___________________ test_ref_count_str ____________________

    def test_ref_count_str():
        y = 'very long and probbably very unique string'
>       assert sys.getrefcount(y) == 2
E       AssertionError: assert 3 == 2
E        +  where 3 = <built-in function getrefcount>('very long and probbably very unique string')
E        +    where <built-in function getrefcount> = sys.getrefcount

stackoverflow_test.py:11: AssertionError

Why 3 and not 2?

Question: How come that

• python = 4 ref counts
• pytest = 3 ref counts
• ipython session = 2 ref counts

I would expect it to behave like in ipython in all 3 cases, can anybody explain what is going on, and give me some hints how to best test the objects I am creating.

Answer 1

Literals in your code are stored in a code object. The bytecode stack is another reference:

>>> import dis
>>> def normal_te_st():
...     x = 222677461248192962146784178
...     y = '!!!!very long and probbably unique string!!!!'
...     print ('x refcount = {}'.format(sys.getrefcount(x)))
...     print ('y refcount = {}'.format(sys.getrefcount(y)))
...
>>> normal_te_st.__code__.co_consts
(None, 222677461248192962146784178, '!!!!very long and probbably unique string!!!!', 'x refcount = {}', 'y refcount = {}')
>>> dis.dis(normal_te_st)
  2           0 LOAD_CONST               1 (222677461248192962146784178)
              2 STORE_FAST               0 (x)

  3           4 LOAD_CONST               2 ('!!!!very long and probbably unique string!!!!')
              6 STORE_FAST               1 (y)

  4           8 LOAD_GLOBAL              0 (print)
             10 LOAD_CONST               3 ('x refcount = {}')
             12 LOAD_ATTR                1 (format)
             14 LOAD_GLOBAL              2 (sys)
             16 LOAD_ATTR                3 (getrefcount)
             18 LOAD_FAST                0 (x)
             20 CALL_FUNCTION            1
             22 CALL_FUNCTION            1
             24 CALL_FUNCTION            1
             26 POP_TOP

  5          28 LOAD_GLOBAL              0 (print)
             30 LOAD_CONST               4 ('y refcount = {}')
             32 LOAD_ATTR                1 (format)
             34 LOAD_GLOBAL              2 (sys)
             36 LOAD_ATTR                3 (getrefcount)
             38 LOAD_FAST                1 (y)
             40 CALL_FUNCTION            1
             42 CALL_FUNCTION            1
             44 CALL_FUNCTION            1
             46 POP_TOP
             48 LOAD_CONST               0 (None)
             50 RETURN_VALUE

The LOAD_CONST opcodes load the object from the co_consts tuple attached to the code object; that tuple is one reference. STORE_FAST then puts that into a local variable, that's the second reference.

Then there's the LOAD_FAST opcode, this takes a name from local storage and puts it on the stack, again incrementing the reference count .

Last but not least, you pass that value to the sys.getrefcount() call.

If you want to learn about what references your objects, you may want to look at gc.get_referrers() ; this function excludes itself and the stack when called, so you can mentally add +2:

>>> import gc
>>> def gc_demo():
...     x = 222677461248192962146784178
...     print(gc.get_referrers(x))
...
>>> gc_demo()
[(None, 222677461248192962146784178), <frame object at 0x106a25a98>]

That prints 2 objects; the co_consts tuple, and the current call frame (for the locals).

py.test does some additional import-time magic which rewrites assert statements , and as a result the reference count is different again.

You may also want to read the Reference Counts section of the Extending Python with C or C++ documentation, the Objects, Types and Reference Counts section of the C API Reference Manual , and last but not least the Debugging Builds section of the same same, to learn how to create a Python build that helps you trace reference counts in detail.

You should never rely on a specific number of references to an object. I can trivially add more references to you objects by reaching into the function object, for example ( foo = normal_te_st.__code__.co_conts[1] would increment the reference count before even running the function). What exactly requires the reference count to go up is an implementation detail. Just make sure your own code handles references correctly.