Memory leak in Python-script using VTK

Question

I am currently using a Python script to process information stored in the EnSight Gold format. My Python (2.6) scipt uses VTK (5.10.0) to process the file, where I used the vtkEnSightGoldReader for reading the data, and loop over time steps. In principle this works find for smaller datasets, however, for large datasets (GBs), I see the memory usage (via top ) increasing with time while the process is running. This filling of the memory goes slow, but in some cases problems are inevitable.

The following script is the minimal productive script that I reduced my issue to.

import vtk

reader = vtk.vtkEnSightGoldReader()

reader.SetCaseFileName("case.case")
reader.Update()

# Get time values
timeset=reader.GetTimeSets()
time=timeset.GetItem(0)
timesteps=time.GetSize()

#reader.ReleaseDataFlagOn()

for j in range(timesteps):
    curTime=time.GetTuple(j)[0]
    print curTime
    reader.SetTimeValue(curTime)
    reader.Update()

    #reader.RemoveAllInputs()

My question is, how can I unload/replace the data that is stored in the memory, instead of using more memory continuously?

As you can see in my source code, I tried member functions "RemoveAllInputs" and "ReleaseDataFlagOn", but they don't work or I used them in the wrong way. Infortunately, I am not getting any closer to a solution.

Something else I tried is the DeepCopy() approach, which I found on the VTK website . However, it seems that this approach is not useful for me, because I get the memory issues even before calling GetOutput()

Answer 1

There is indeed a (minor) memory leak in the vtkEnsightGoldReader. The memory leak is a result of not properly clear collection object, which becomes apparent only for processing very large datasets. Technically it is not a memoryleak, since it gets properly cleared after a run.

This can only be solved by applying a patch to the VTK source and recompiling. I received the patch below via people from Kitware, so I would assume this rolled out in later versions of VTK.

diff --git a/IO/vtkEnSightReader.cxx b/IO/vtkEnSightReader.cxx
index 68a9b8f..7ab8ddd 100644
--- a/IO/vtkEnSightReader.cxx
+++ b/IO/vtkEnSightReader.cxx
@@ -985,6 +985,8 @@ int vtkEnSightReader::ReadCaseFileTime(char* line)
   int timeSet, numTimeSteps, i, filenameNum, increment, lineRead;
   float timeStep;

+  this->TimeSetFileNameNumbers->RemoveAllItems();
+
   // found TIME section
   int firstTimeStep = 1;