C＃多线程对表的列性能问题

Question

I am currently having an issue with getting the performance benefit using C# Threads. What I am currently doing here is encrypting the entire contents of selected columns in a tabular file (.csv). The program is used for generally large files, whose size may reach up to terabytes in size, with millions of rows and numerous columns.

In order to achieve optimal performance, I planned to create and run the processing algorithm in separate threads for each column. I believe the sheer amount of computation needed for each column warrants a thread each. Or at least it was true when I was doing similar projects using C++ threads .

Now, for some reason, the code snippet below DOES NOT produce any speed-up what so ever. In fact, the time taken compared to single thread sequential processing differs in only a few seconds SLOWER . It is roughly the same result whether I'm processing only 1 column, 4 columns, or 128 columns simultaneously.

// NOTE:
// m_TableData is of type |--Dictionary<int, List<string>>--|
// Key   == Column Number
// Value == Column Contents

List<Thread> Threads = new List<Thread>();

// encrypt data in selected columns
foreach (var KeyPair in m_TableData)
{
  Threads.Add(new Thread(new ThreadStart(() =>
  {
    // Process each row element
    // NOTE: ColSize is usually huge (>10,000)
    int ColSize = KeyPair.Value.Count();
    for (int i = 0; i < ColSize; ++i)
    {
      m_TableData[KeyPair.Key][i] = ProcessingAlgorithm(m_TableData[KeyPair.Key][i]);
    }
  })));
  Threads.Last().Start();
}

foreach (Thread th in Threads)
  th.Join();

In hopes of trying to prevent False Sharing and/or better cache performance I have even tried splitting up the rows into batches of 500, 1000, 10000, etc, but to no avail. I have even tried the System.Threading.Tasks.Parallel.ForEach function, and it also seems to give the same results. The lack of performance improvement has me really scratching my head.

Any professional/experienced parallel-processing programmers over here? I greatly appreciate any and all feedback and criticism on my code and issue. Thanks!

Answer 1

Pretty sure this isn't your biggest performance issue - something else is going on here (really need to see what your ProcessingAlgorithm is doing) - but you can eliminate 2 dictionary lookups in your loop by replacing:

m_TableData[KeyPair.Key]

with

KeyPair.Value

So you end up with:

for (int i = 0; i < ColSize; ++i)
{
    KeyPair.Value[i] = ProcessingAlgorithm(KeyPair.Value[i]);
}

(I'd probably actually assign the list to a local variable instead of using the KeyPair)