数据流管道保持在内存中

Question

I've created a Dataflow pipeline consisting of 4 blocks (which includes one optional block) which is responsible for receiving a query object from my application across HTTP and retrieving information from a database, doing an optional transform on that data, and then writing the information back in the HTTP response.

In some testing I've done I've been pulling down a significant amount of data from the database (570 thousand rows) which are stored in a List object and passed between the different blocks and it seems like even after the final block has been completed the memory isn't being released. Ram usage in Task Manager will spike up to over 2 GB and I can observe several large spikes as the List hits each block.

The signatures for my blocks look like this:

private TransformBlock<HttpListenerContext, Tuple<HttpListenerContext, QueryObject>> m_ParseHttpRequest;
private TransformBlock<Tuple<HttpListenerContext, QueryObject>, Tuple<HttpListenerContext, QueryObject, List<string>>> m_RetrieveDatabaseResults;
private TransformBlock<Tuple<HttpListenerContext, QueryObject, List<string>>, Tuple<HttpListenerContext, QueryObject, List<string>>> m_ConvertResults;
private ActionBlock<Tuple<HttpListenerContext, QueryObject, List<string>>> m_ReturnHttpResponse;

They are linked as follows:

m_ParseHttpRequest.LinkTo(m_RetrieveDatabaseResults);
m_RetrieveDatabaseResults.LinkTo(m_ConvertResults, tuple => tuple.Item2 is QueryObjectA);   
m_RetrieveDatabaseResults.LinkTo(m_ReturnHttpResponse, tuple => tuple.Item2 is QueryObjectB);           
m_ConvertResults.LinkTo(m_ReturnHttpResponse);

Is it possible that I can set up the pipeline such that once each block is done with the list they no longer need to hold on to it as well as once the entire pipeline is completed that the memory is released?

Answer 1

1. There is no reason why dataflow blocks should hold references to already processed messages. Because of that, I believe they don't do that. If they actually do, then I think that's a bug that you should report.

2. Big RAM usage reported in Task Manager does not necessarily mean the objects weren't collected. It could also mean that the framework didn't return the GCed memory to the OS yet. I believe that when you have enough free memory, the framework is quite lazy about returning memory, because there is no good reason to do so.

   What you should do instead is to use more appropriate tools. For example, you can use .Net performance counters to monitor the GC quite closely. You could also use a memory profiler to see more detailed information about memory usage and, if you actually have a memory leak, what causes it.