繁体   English   中英

使用 boto3 时垃圾收集不正常

[英]Garbage collection is not happening properly when using boto3

我只是运行代码以循环收听 sqs 消息 n 次。 每次迭代后,我都调用gc.collect()但它返回了一些无法访问的对象,而且我正在检查gc.garbagge中是否有gc未收集的对象,此列表也随着每次迭代而不断增加。

示例代码:

import os
import gc
import boto3
import psutil

gc.set_debug(gc.DEBUG_SAVEALL | gc.DEBUG_UNCOLLECTABLE)
def get_memory_usage():
    return psutil.Process(os.getpid()).memory_info().rss // 1024 ** 2


def test():
    queue_url = 'https://sqs.us-east-2.amazonaws.com/123/test.fifo'
    sqs = create_client('sqs')
    for i in range(250):
        message = sqs.receive_message(QueueUrl=queue_url)
        if message.get('Messages'):
            recept_handle = message['Messages'][0]['ReceiptHandle']
            sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=recept_handle)

        print(f'Iteration - {i + 1} Unreachable Objects: {gc.collect()} and length: {len(gc.garbage)}')


print(f'Memory usage Before: {get_memory_usage()}mb')
test()
print(f'==================Unreachable Objects: {gc.collect()}==================')
print(len(gc.garbage))
print(f'Memory usage After: {get_memory_usage()}mb')

Output:

Memory usage Before: 27mb
Iteration - 1 Unreachable Objects: 449 and length: 554
Iteration - 2 Unreachable Objects: 3 and length: 557
Iteration - 3 Unreachable Objects: 3 and length: 560
Iteration - 4 Unreachable Objects: 3 and length: 563
Iteration - 5 Unreachable Objects: 3 and length: 566
Iteration - 6 Unreachable Objects: 3 and length: 569
Iteration - 7 Unreachable Objects: 3 and length: 572
Iteration - 8 Unreachable Objects: 3 and length: 575
Iteration - 9 Unreachable Objects: 3 and length: 578
Iteration - 10 Unreachable Objects: 3 and length: 581
Iteration - 11 Unreachable Objects: 3 and length: 584
Iteration - 12 Unreachable Objects: 3 and length: 587
Iteration - 13 Unreachable Objects: 3 and length: 590
Iteration - 14 Unreachable Objects: 3 and length: 593
Iteration - 15 Unreachable Objects: 3 and length: 596
Iteration - 16 Unreachable Objects: 3 and length: 599
Iteration - 17 Unreachable Objects: 3 and length: 602
Iteration - 18 Unreachable Objects: 3 and length: 605
Iteration - 19 Unreachable Objects: 3 and length: 608
Iteration - 20 Unreachable Objects: 3 and length: 611
Iteration - 21 Unreachable Objects: 3 and length: 614
Iteration - 22 Unreachable Objects: 3 and length: 617
Iteration - 23 Unreachable Objects: 3 and length: 620
Iteration - 24 Unreachable Objects: 3 and length: 623
Iteration - 25 Unreachable Objects: 3 and length: 626
Iteration - 26 Unreachable Objects: 3 and length: 629
Iteration - 27 Unreachable Objects: 3 and length: 632
Iteration - 28 Unreachable Objects: 3 and length: 635
Iteration - 29 Unreachable Objects: 3 and length: 638
Iteration - 30 Unreachable Objects: 3 and length: 641
Iteration - 31 Unreachable Objects: 3 and length: 644
Iteration - 32 Unreachable Objects: 3 and length: 647
Iteration - 33 Unreachable Objects: 3 and length: 650
Iteration - 34 Unreachable Objects: 3 and length: 653
Iteration - 35 Unreachable Objects: 3 and length: 656
Iteration - 36 Unreachable Objects: 3 and length: 659
Iteration - 37 Unreachable Objects: 3 and length: 662
Iteration - 38 Unreachable Objects: 3 and length: 665
Iteration - 39 Unreachable Objects: 3 and length: 668
Iteration - 40 Unreachable Objects: 3 and length: 671
Iteration - 41 Unreachable Objects: 3 and length: 674
Iteration - 42 Unreachable Objects: 3 and length: 677
Iteration - 43 Unreachable Objects: 3 and length: 680
Iteration - 44 Unreachable Objects: 3 and length: 683
Iteration - 45 Unreachable Objects: 3 and length: 686
Iteration - 46 Unreachable Objects: 3 and length: 689
Iteration - 47 Unreachable Objects: 3 and length: 692
Iteration - 48 Unreachable Objects: 3 and length: 695
Iteration - 49 Unreachable Objects: 3 and length: 698
Iteration - 50 Unreachable Objects: 3 and length: 701
Iteration - 51 Unreachable Objects: 3 and length: 704
Iteration - 52 Unreachable Objects: 3 and length: 707
Iteration - 53 Unreachable Objects: 3 and length: 710
Iteration - 54 Unreachable Objects: 3 and length: 713
Iteration - 55 Unreachable Objects: 3 and length: 716
Iteration - 56 Unreachable Objects: 3 and length: 719
Iteration - 57 Unreachable Objects: 3 and length: 722
Iteration - 58 Unreachable Objects: 3 and length: 725
Iteration - 59 Unreachable Objects: 3 and length: 728
Iteration - 60 Unreachable Objects: 3 and length: 731
Iteration - 61 Unreachable Objects: 3 and length: 734
Iteration - 62 Unreachable Objects: 3 and length: 737
Iteration - 63 Unreachable Objects: 3 and length: 740
Iteration - 64 Unreachable Objects: 3 and length: 743
Iteration - 65 Unreachable Objects: 3 and length: 746
Iteration - 66 Unreachable Objects: 3 and length: 749
Iteration - 67 Unreachable Objects: 3 and length: 752
Iteration - 68 Unreachable Objects: 3 and length: 755
Iteration - 69 Unreachable Objects: 3 and length: 758
Iteration - 70 Unreachable Objects: 3 and length: 761
Iteration - 71 Unreachable Objects: 3 and length: 764
Iteration - 72 Unreachable Objects: 3 and length: 767
Iteration - 73 Unreachable Objects: 3 and length: 770
Iteration - 74 Unreachable Objects: 3 and length: 773
Iteration - 75 Unreachable Objects: 3 and length: 776
Iteration - 76 Unreachable Objects: 3 and length: 779
Iteration - 77 Unreachable Objects: 3 and length: 782
Iteration - 78 Unreachable Objects: 3 and length: 785
Iteration - 79 Unreachable Objects: 3 and length: 788
Iteration - 80 Unreachable Objects: 3 and length: 791
Iteration - 81 Unreachable Objects: 3 and length: 794
Iteration - 82 Unreachable Objects: 3 and length: 797
Iteration - 83 Unreachable Objects: 3 and length: 800
Iteration - 84 Unreachable Objects: 3 and length: 803
Iteration - 85 Unreachable Objects: 3 and length: 806
Iteration - 86 Unreachable Objects: 3 and length: 809
Iteration - 87 Unreachable Objects: 3 and length: 812
Iteration - 88 Unreachable Objects: 3 and length: 815
Iteration - 89 Unreachable Objects: 3 and length: 818
Iteration - 90 Unreachable Objects: 3 and length: 821
Iteration - 91 Unreachable Objects: 3 and length: 824
Iteration - 92 Unreachable Objects: 3 and length: 827
Iteration - 93 Unreachable Objects: 3 and length: 830
Iteration - 94 Unreachable Objects: 3 and length: 833
Iteration - 95 Unreachable Objects: 3 and length: 836
Iteration - 96 Unreachable Objects: 3 and length: 839
Iteration - 97 Unreachable Objects: 3 and length: 842
Iteration - 98 Unreachable Objects: 3 and length: 845
Iteration - 99 Unreachable Objects: 3 and length: 848
Iteration - 100 Unreachable Objects: 3 and length: 851
Iteration - 101 Unreachable Objects: 3 and length: 854
Iteration - 102 Unreachable Objects: 3 and length: 857
Iteration - 103 Unreachable Objects: 3 and length: 860
Iteration - 104 Unreachable Objects: 3 and length: 863
Iteration - 105 Unreachable Objects: 3 and length: 866
Iteration - 106 Unreachable Objects: 3 and length: 869
Iteration - 107 Unreachable Objects: 3 and length: 872
Iteration - 108 Unreachable Objects: 3 and length: 875
Iteration - 109 Unreachable Objects: 3 and length: 878
Iteration - 110 Unreachable Objects: 3 and length: 881
Iteration - 111 Unreachable Objects: 3 and length: 884
Iteration - 112 Unreachable Objects: 3 and length: 887
Iteration - 113 Unreachable Objects: 3 and length: 890
Iteration - 114 Unreachable Objects: 3 and length: 893
Iteration - 115 Unreachable Objects: 3 and length: 896
Iteration - 116 Unreachable Objects: 3 and length: 899
Iteration - 117 Unreachable Objects: 3 and length: 902
Iteration - 118 Unreachable Objects: 3 and length: 905
Iteration - 119 Unreachable Objects: 3 and length: 908
Iteration - 120 Unreachable Objects: 3 and length: 911
Iteration - 121 Unreachable Objects: 3 and length: 914
Iteration - 122 Unreachable Objects: 3 and length: 917
Iteration - 123 Unreachable Objects: 3 and length: 920
Iteration - 124 Unreachable Objects: 3 and length: 923
Iteration - 125 Unreachable Objects: 3 and length: 926
Iteration - 126 Unreachable Objects: 3 and length: 929
Iteration - 127 Unreachable Objects: 3 and length: 932
Iteration - 128 Unreachable Objects: 3 and length: 935
Iteration - 129 Unreachable Objects: 3 and length: 938
Iteration - 130 Unreachable Objects: 3 and length: 941
Iteration - 131 Unreachable Objects: 3 and length: 944
Iteration - 132 Unreachable Objects: 3 and length: 947
Iteration - 133 Unreachable Objects: 3 and length: 950
Iteration - 134 Unreachable Objects: 3 and length: 953
Iteration - 135 Unreachable Objects: 3 and length: 956
Iteration - 136 Unreachable Objects: 3 and length: 959
Iteration - 137 Unreachable Objects: 3 and length: 962
Iteration - 138 Unreachable Objects: 3 and length: 965
Iteration - 139 Unreachable Objects: 3 and length: 968
Iteration - 140 Unreachable Objects: 3 and length: 971
Iteration - 141 Unreachable Objects: 3 and length: 974
Iteration - 142 Unreachable Objects: 3 and length: 977
Iteration - 143 Unreachable Objects: 3 and length: 980
Iteration - 144 Unreachable Objects: 3 and length: 983
Iteration - 145 Unreachable Objects: 3 and length: 986
Iteration - 146 Unreachable Objects: 3 and length: 989
Iteration - 147 Unreachable Objects: 3 and length: 992
Iteration - 148 Unreachable Objects: 3 and length: 995
Iteration - 149 Unreachable Objects: 3 and length: 998
Iteration - 150 Unreachable Objects: 3 and length: 1001
Iteration - 151 Unreachable Objects: 3 and length: 1004
Iteration - 152 Unreachable Objects: 3 and length: 1007
Iteration - 153 Unreachable Objects: 3 and length: 1010
Iteration - 154 Unreachable Objects: 3 and length: 1013
Iteration - 155 Unreachable Objects: 3 and length: 1016
Iteration - 156 Unreachable Objects: 3 and length: 1019
Iteration - 157 Unreachable Objects: 3 and length: 1022
Iteration - 158 Unreachable Objects: 3 and length: 1025
Iteration - 159 Unreachable Objects: 3 and length: 1028
Iteration - 160 Unreachable Objects: 3 and length: 1031
Iteration - 161 Unreachable Objects: 3 and length: 1034
Iteration - 162 Unreachable Objects: 3 and length: 1037
Iteration - 163 Unreachable Objects: 3 and length: 1040
Iteration - 164 Unreachable Objects: 3 and length: 1043
Iteration - 165 Unreachable Objects: 3 and length: 1046
Iteration - 166 Unreachable Objects: 3 and length: 1049
Iteration - 167 Unreachable Objects: 3 and length: 1052
Iteration - 168 Unreachable Objects: 3 and length: 1055
Iteration - 169 Unreachable Objects: 3 and length: 1058
Iteration - 170 Unreachable Objects: 3 and length: 1061
Iteration - 171 Unreachable Objects: 3 and length: 1064
Iteration - 172 Unreachable Objects: 3 and length: 1067
Iteration - 173 Unreachable Objects: 3 and length: 1070
Iteration - 174 Unreachable Objects: 3 and length: 1073
Iteration - 175 Unreachable Objects: 3 and length: 1076
Iteration - 176 Unreachable Objects: 3 and length: 1079
Iteration - 177 Unreachable Objects: 3 and length: 1082
Iteration - 178 Unreachable Objects: 3 and length: 1085
Iteration - 179 Unreachable Objects: 3 and length: 1088
Iteration - 180 Unreachable Objects: 3 and length: 1091
Iteration - 181 Unreachable Objects: 3 and length: 1094
Iteration - 182 Unreachable Objects: 3 and length: 1097
Iteration - 183 Unreachable Objects: 3 and length: 1100
Iteration - 184 Unreachable Objects: 3 and length: 1103
Iteration - 185 Unreachable Objects: 3 and length: 1106
Iteration - 186 Unreachable Objects: 3 and length: 1109
Iteration - 187 Unreachable Objects: 3 and length: 1112
Iteration - 188 Unreachable Objects: 3 and length: 1115
Iteration - 189 Unreachable Objects: 3 and length: 1118
Iteration - 190 Unreachable Objects: 3 and length: 1121
Iteration - 191 Unreachable Objects: 3 and length: 1124
Iteration - 192 Unreachable Objects: 3 and length: 1127
Iteration - 193 Unreachable Objects: 3 and length: 1130
Iteration - 194 Unreachable Objects: 3 and length: 1133
Iteration - 195 Unreachable Objects: 3 and length: 1136
Iteration - 196 Unreachable Objects: 3 and length: 1139
Iteration - 197 Unreachable Objects: 3 and length: 1142
Iteration - 198 Unreachable Objects: 3 and length: 1145
Iteration - 199 Unreachable Objects: 3 and length: 1148
Iteration - 200 Unreachable Objects: 3 and length: 1151
Iteration - 201 Unreachable Objects: 3 and length: 1154
Iteration - 202 Unreachable Objects: 3 and length: 1157
Iteration - 203 Unreachable Objects: 3 and length: 1160
Iteration - 204 Unreachable Objects: 3 and length: 1163
Iteration - 205 Unreachable Objects: 3 and length: 1166
Iteration - 206 Unreachable Objects: 3 and length: 1169
Iteration - 207 Unreachable Objects: 3 and length: 1172
Iteration - 208 Unreachable Objects: 3 and length: 1175
Iteration - 209 Unreachable Objects: 3 and length: 1178
Iteration - 210 Unreachable Objects: 3 and length: 1181
Iteration - 211 Unreachable Objects: 3 and length: 1184
Iteration - 212 Unreachable Objects: 3 and length: 1187
Iteration - 213 Unreachable Objects: 3 and length: 1190
Iteration - 214 Unreachable Objects: 3 and length: 1193
Iteration - 215 Unreachable Objects: 3 and length: 1196
Iteration - 216 Unreachable Objects: 3 and length: 1199
Iteration - 217 Unreachable Objects: 3 and length: 1202
Iteration - 218 Unreachable Objects: 3 and length: 1205
Iteration - 219 Unreachable Objects: 3 and length: 1208
Iteration - 220 Unreachable Objects: 3 and length: 1211
Iteration - 221 Unreachable Objects: 3 and length: 1214
Iteration - 222 Unreachable Objects: 3 and length: 1217
Iteration - 223 Unreachable Objects: 3 and length: 1220
Iteration - 224 Unreachable Objects: 3 and length: 1223
Iteration - 225 Unreachable Objects: 3 and length: 1226
Iteration - 226 Unreachable Objects: 3 and length: 1229
Iteration - 227 Unreachable Objects: 3 and length: 1232
Iteration - 228 Unreachable Objects: 3 and length: 1235
Iteration - 229 Unreachable Objects: 3 and length: 1238
Iteration - 230 Unreachable Objects: 3 and length: 1241
Iteration - 231 Unreachable Objects: 3 and length: 1244
Iteration - 232 Unreachable Objects: 3 and length: 1247
Iteration - 233 Unreachable Objects: 3 and length: 1250
Iteration - 234 Unreachable Objects: 3 and length: 1253
Iteration - 235 Unreachable Objects: 3 and length: 1256
Iteration - 236 Unreachable Objects: 3 and length: 1259
Iteration - 237 Unreachable Objects: 3 and length: 1262
Iteration - 238 Unreachable Objects: 3 and length: 1265
Iteration - 239 Unreachable Objects: 3 and length: 1268
Iteration - 240 Unreachable Objects: 3 and length: 1271
Iteration - 241 Unreachable Objects: 3 and length: 1274
Iteration - 242 Unreachable Objects: 3 and length: 1277
Iteration - 243 Unreachable Objects: 3 and length: 1280
Iteration - 244 Unreachable Objects: 3 and length: 1283
Iteration - 245 Unreachable Objects: 3 and length: 1286
Iteration - 246 Unreachable Objects: 3 and length: 1289
Iteration - 247 Unreachable Objects: 3 and length: 1292
Iteration - 248 Unreachable Objects: 3 and length: 1295
Iteration - 249 Unreachable Objects: 3 and length: 1298
Iteration - 250 Unreachable Objects: 3 and length: 1301
==================Unreachable Objects: 233==================
1534
Memory usage After: 37mb

这只是一个示例代码,在我的实际用例中,我在一个单独的线程中以无限循环运行代码,因此 memory 一直在累积。

请帮助我理解这里的问题是正常的还是我的代码有问题

前言:

这里似乎有两个不同的问题——第一个是关于在线程中使用boto3“在我的实际用例中,我在一个单独的线程中以无限循环运行代码” ),另一个是询问问题提供的示例代码。

解决第一个问题; 据报道,在某些实现模式中,boto3 的Session()对象可能与线程“不能很好地配合”:

  • 客户永远不会被 GC #805
  • memory 在多线程上使用过多#1670

由于示例代码根本没有演示线程,因此很难判断是否属于这种情况。 出于这个原因,我将使用提供的示例代码解决“第二个”问题: “is something %s”

如果这不能回答问题,那么提供一个演示线程问题的示例代码将非常有帮助。


如果gc.collect()返回一个正数,那不一定表示有问题。 这只是意味着收集了具有正引用计数的对象,因为对它们的引用都不可访问。
至于 memory 使用量的增加,这是因为设置了 gc.DEBUG_SAVEALL 此设置将阻止实际的 object 释放,因为它们被添加到gc.garbage列表中,并且无法释放它们使用的 memory。

为了演示,以下代码大致基于所提供的示例:

import gc
import os
from contextlib import contextmanager
import psutil

ITERATIONS = 10
SIZE = 1024 * 1024     # 1MB

gc.set_debug(gc.DEBUG_SAVEALL | gc.DEBUG_UNCOLLECTABLE)

def get_memory_usage():
    return psutil.Process(os.getpid()).memory_info().rss // 1024 ** 2

@contextmanager
def print_ps_mem_usage(subtext):
    print(f"{'->' * 2} Memory before '{subtext}': {get_memory_usage()}mb {'->' * 2}")
    try:
        yield
    finally:
        print(f"  <- Memory after  '{subtext}': {get_memory_usage()}mb <-")

class MemContainer:
    def __init__(self, hogger, size):
        # Create a reference to MemHogger instance, facilitating a reference loop
        self.hogger = hogger
        self.megabyte = (f"{self.hogger.index} " * size)[:size]

    def __str__(self):
        return f"{self.__class__.__name__}({len(self.megabyte)/SIZE}mb)->{self.hogger!s}"

class MemHogger:
    def __init__(self, index, size):
        self.index = index
        # By keeping a link to the newly created MemContainer, we are creating a reference loop
        self.container = MemContainer(self, size)

    def __str__(self):
        return f"{self.__class__.__name__}({self.index})"

def loop_hogger():
    for i in range(ITERATIONS):
        with print_ps_mem_usage(f"Creating MemHogger {i}"):
            mh = MemHogger(i, SIZE)
        unreachable = gc.collect()
        unreachable_types = [type(unreach) for unreach in gc.garbage[-unreachable:]]
        unreachable_contents = [list(unreach.keys()) if isinstance(unreach, dict) else str(unreach) for unreach in gc.garbage[-unreachable:]]
        print(f'Iteration - {i + 1} Unreachable Objects: {unreachable} and length: {len(gc.garbage)}, items: {list(zip(unreachable_types, unreachable_contents))}')


with print_ps_mem_usage("main run"):
    loop_hogger()
    print(f'==================Unreachable Objects: {gc.collect()}==================')
    print(len(gc.garbage))

一般流程:

  • MemHogger存储索引(当前迭代)和对MemContainer实例的引用。 MemContainer实例存储与索引相关的 ~1MB 字符串,以及对调用MemHogger实例的引用。 这会在两者之间创建一个引用循环,这会占用大约 1MB 的 RAM。
  • 第一次迭代将mh声明为MemHogger(0) ,在该迭代结束时仍然可以访问; 收集了 0 个对象。
  • 后续迭代将mh重置为当前i ,因此无法再通过运行代码访问在上一次迭代中创建的实例。
  • 无法访问的MemHogger实例(来自上一次迭代)仍由其MemContainer实例引用(refcount>0),因此不会立即收集。
  • 在第二次迭代结束时,收集了 4 个无法访问的项(来自上一次迭代):MemHogger、MemHogger 的__dict__ 、MemContainer 和 MemContainer 的__dict__ 它们可以相互访问(引用循环),但不能再通过运行代码访问。
  • 在最后一次迭代中, mh被设置为MemHogger(9) ,但是一旦 function 返回, mh就不再可达,因此上层 scope 收集了另外 4 个无法到达的项目。

output 应如下所示:

->-> Memory before 'main run': 10mb ->->
->-> Memory before 'Creating MemHogger 0': 10mb ->->
  <- Memory after  'Creating MemHogger 0': 11mb <-
Iteration - 1 Unreachable Objects: 0 and length: 0, items: []
->-> Memory before 'Creating MemHogger 1': 11mb ->->
  <- Memory after  'Creating MemHogger 1': 14mb <-
Iteration - 2 Unreachable Objects: 4 and length: 4, items: [(<class '__main__.MemHogger'>, 'MemHogger(0)'), (<class 'dict'>, ['index', 'container']), (<class '__main__.MemContainer'>, 'MemContainer(1.0mb)->MemHogger(0)'), (<class 'dict'>, ['hogger', 'megabyte'])]
->-> Memory before 'Creating MemHogger 2': 14mb ->->
  <- Memory after  'Creating MemHogger 2': 16mb <-
Iteration - 3 Unreachable Objects: 4 and length: 8, items: [(<class '__main__.MemHogger'>, 'MemHogger(1)'), (<class 'dict'>, ['index', 'container']), (<class '__main__.MemContainer'>, 'MemContainer(1.0mb)->MemHogger(1)'), (<class 'dict'>, ['hogger', 'megabyte'])]
->-> Memory before 'Creating MemHogger 3': 16mb ->->
  <- Memory after  'Creating MemHogger 3': 17mb <-
Iteration - 4 Unreachable Objects: 4 and length: 12, items: [(<class '__main__.MemHogger'>, 'MemHogger(2)'), (<class 'dict'>, ['index', 'container']), (<class '__main__.MemContainer'>, 'MemContainer(1.0mb)->MemHogger(2)'), (<class 'dict'>, ['hogger', 'megabyte'])]
->-> Memory before 'Creating MemHogger 4': 17mb ->->
  <- Memory after  'Creating MemHogger 4': 18mb <-
Iteration - 5 Unreachable Objects: 4 and length: 16, items: [(<class '__main__.MemHogger'>, 'MemHogger(3)'), (<class 'dict'>, ['index', 'container']), (<class '__main__.MemContainer'>, 'MemContainer(1.0mb)->MemHogger(3)'), (<class 'dict'>, ['hogger', 'megabyte'])]
->-> Memory before 'Creating MemHogger 5': 18mb ->->
  <- Memory after  'Creating MemHogger 5': 19mb <-
Iteration - 6 Unreachable Objects: 4 and length: 20, items: [(<class '__main__.MemHogger'>, 'MemHogger(4)'), (<class 'dict'>, ['index', 'container']), (<class '__main__.MemContainer'>, 'MemContainer(1.0mb)->MemHogger(4)'), (<class 'dict'>, ['hogger', 'megabyte'])]
->-> Memory before 'Creating MemHogger 6': 19mb ->->
  <- Memory after  'Creating MemHogger 6': 20mb <-
Iteration - 7 Unreachable Objects: 4 and length: 24, items: [(<class '__main__.MemHogger'>, 'MemHogger(5)'), (<class 'dict'>, ['index', 'container']), (<class '__main__.MemContainer'>, 'MemContainer(1.0mb)->MemHogger(5)'), (<class 'dict'>, ['hogger', 'megabyte'])]
->-> Memory before 'Creating MemHogger 7': 20mb ->->
  <- Memory after  'Creating MemHogger 7': 21mb <-
Iteration - 8 Unreachable Objects: 4 and length: 28, items: [(<class '__main__.MemHogger'>, 'MemHogger(6)'), (<class 'dict'>, ['index', 'container']), (<class '__main__.MemContainer'>, 'MemContainer(1.0mb)->MemHogger(6)'), (<class 'dict'>, ['hogger', 'megabyte'])]
->-> Memory before 'Creating MemHogger 8': 21mb ->->
  <- Memory after  'Creating MemHogger 8': 22mb <-
Iteration - 9 Unreachable Objects: 4 and length: 32, items: [(<class '__main__.MemHogger'>, 'MemHogger(7)'), (<class 'dict'>, ['index', 'container']), (<class '__main__.MemContainer'>, 'MemContainer(1.0mb)->MemHogger(7)'), (<class 'dict'>, ['hogger', 'megabyte'])]
->-> Memory before 'Creating MemHogger 9': 22mb ->->
  <- Memory after  'Creating MemHogger 9': 23mb <-
Iteration - 10 Unreachable Objects: 4 and length: 36, items: [(<class '__main__.MemHogger'>, 'MemHogger(8)'), (<class 'dict'>, ['index', 'container']), (<class '__main__.MemContainer'>, 'MemContainer(1.0mb)->MemHogger(8)'), (<class 'dict'>, ['hogger', 'megabyte'])]
==================Unreachable Objects: 4==================
40
  <- Memory after  'main run': 23mb <-

虽然rss还考虑了 python 的内部 memory 分配,但可以观察到,在大多数迭代中, rss以 1MB 的增量增加,因为这(大致)是 MemContainer 的实例大小。

当排除 SAVEALL 设置(即gc.set_debug(gc.DEBUG_UNCOLLECTABLE) )时,无法访问的对象被释放,因此虽然仍然找到 4 个对象, gc.garbage仍然为空,并且 output 显示rss没有显着增加:

->-> Memory before 'main run': 10mb ->->
->-> Memory before 'Creating MemHogger 0': 10mb ->->
  <- Memory after  'Creating MemHogger 0': 11mb <-
Iteration - 1 Unreachable Objects: 0 and length: 0, items: []
->-> Memory before 'Creating MemHogger 1': 11mb ->->
  <- Memory after  'Creating MemHogger 1': 14mb <-
Iteration - 2 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 2': 13mb ->->
  <- Memory after  'Creating MemHogger 2': 15mb <-
Iteration - 3 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 3': 12mb ->->
  <- Memory after  'Creating MemHogger 3': 14mb <-
Iteration - 4 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 4': 14mb ->->
  <- Memory after  'Creating MemHogger 4': 14mb <-
Iteration - 5 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 5': 14mb ->->
  <- Memory after  'Creating MemHogger 5': 14mb <-
Iteration - 6 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 6': 14mb ->->
  <- Memory after  'Creating MemHogger 6': 14mb <-
Iteration - 7 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 7': 14mb ->->
  <- Memory after  'Creating MemHogger 7': 14mb <-
Iteration - 8 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 8': 14mb ->->
  <- Memory after  'Creating MemHogger 8': 14mb <-
Iteration - 9 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 9': 14mb ->->
  <- Memory after  'Creating MemHogger 9': 14mb <-
Iteration - 10 Unreachable Objects: 4 and length: 0, items: []
==================Unreachable Objects: 4==================
0
  <- Memory after  'main run': 14mb <-

设置ITERATIONS = 10000会产生类似的结果:

->-> Memory before 'main run': 10mb ->->
->-> Memory before 'Creating MemHogger 0': 10mb ->->
  <- Memory after  'Creating MemHogger 0': 11mb <-
Iteration - 1 Unreachable Objects: 0 and length: 0, items: []
->-> Memory before 'Creating MemHogger 1': 11mb ->->
  <- Memory after  'Creating MemHogger 1': 14mb <-
Iteration - 2 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 2': 13mb ->->
  <- Memory after  'Creating MemHogger 2': 15mb <-
Iteration - 3 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 3': 11mb ->->
  <- Memory after  'Creating MemHogger 3': 14mb <-
Iteration - 4 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 4': 14mb ->->
  <- Memory after  'Creating MemHogger 4': 14mb <-
Iteration - 5 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 5': 14mb ->->
  <- Memory after  'Creating MemHogger 5': 14mb <-
Iteration - 6 Unreachable Objects: 4 and length: 0, items: []
...
->-> Memory before 'Creating MemHogger 11': 14mb ->->
  <- Memory after  'Creating MemHogger 11': 15mb <-
Iteration - 12 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 12': 15mb ->->
  <- Memory after  'Creating MemHogger 12': 15mb <-
Iteration - 13 Unreachable Objects: 4 and length: 0, items: []
...
->-> Memory before 'Creating MemHogger 101': 15mb ->->
  <- Memory after  'Creating MemHogger 101': 16mb <-
Iteration - 102 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 102': 16mb ->->
  <- Memory after  'Creating MemHogger 102': 16mb <-
Iteration - 103 Unreachable Objects: 4 and length: 0, items: []
...
->-> Memory before 'Creating MemHogger 1001': 16mb ->->
  <- Memory after  'Creating MemHogger 1001': 17mb <-
Iteration - 1002 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 1002': 17mb ->->
  <- Memory after  'Creating MemHogger 1002': 17mb <-
Iteration - 1003 Unreachable Objects: 4 and length: 0, items: []
...
Iteration - 9995 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 9995': 17mb ->->
  <- Memory after  'Creating MemHogger 9995': 17mb <-
Iteration - 9996 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 9996': 17mb ->->
  <- Memory after  'Creating MemHogger 9996': 17mb <-
Iteration - 9997 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 9997': 17mb ->->
  <- Memory after  'Creating MemHogger 9997': 17mb <-
Iteration - 9998 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 9998': 17mb ->->
  <- Memory after  'Creating MemHogger 9998': 17mb <-
Iteration - 9999 Unreachable Objects: 4 and length: 0, items: []
->-> Memory before 'Creating MemHogger 9999': 17mb ->->
  <- Memory after  'Creating MemHogger 9999': 17mb <-
Iteration - 10000 Unreachable Objects: 4 and length: 0, items: []
==================Unreachable Objects: 4==================
0
  <- Memory after  'main run': 17mb <-

暂无
暂无

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM