处理C＃中大型数据数组的最有效方法？

Question

Currently I am using XNA Game Studio 4.0 with C# Visual Studio 2010. I want to use a versatile method for handling triangles. I am using a preset array of VertexPositionColor items passed through the GraphicsDevice.DrawUserPrimitives() method, which only handles arrays. Because arrays are fixed, but I wanted to have a very large space to arbitrarily add new triangles to the array, my original idea was to make a large array, specifically

VertexPositionColor vertices = new VertexPositionColor[int.MaxValue];

but that ran my application out of memory. So what I'm wondering is how to approach this memory/performance issue best.

• Is there an easy way to increase the amount of memory allocated to the stack whenever my program runs?
• Would it be beneficial to store the array on the heap instead? And would I have to build my own allocator if I wanted to do that?
• Or is my best approach simply to use a LinkedList and deal with the extra processing required to copy it to an array every frame?

Answer 1

I hit this building my voxel engine code. Consider the problem I had:

Given an unknown volume size that would clearly be bigger than the amount of memory the computer had how do I manage that volume of data?

My solution was to use sparse chunking. for example:

In my case instead of using an array I used a dictionary. This way I could lookup the values based on a key that was say the hashcode of a voxels position and the value was the voxel itself. This meant that the voxels were fast to pull out, and self organised by the language / compiler in to an indexed set.

It also means that when pulling data back out I could default to Voxel.Empty for voxels that hadn't yet been assigned.

In your case you might not need a default value but using a dictionary might prove more helpful than an array.

The up shot ... Arrays are a tad faster for some things but when you consider all of your usage scenarios for the data you may find that overall the gains of using a dictionary are worth a slight allocation cost.

In testing I found that if I was prepared to drop from something like 100ms per thousand to say 120ms per thousand on allocations I could then retrieve the data 100% faster for most of the queries I was performing on the set.

Reason for my suggestion here:

It looks like you don't know the size of your data set and using an array only makes sense if you do know the size otherwise you tie up needless "pre allocated chunks of ram" for no reason in order to make your code ready for any eventuality you want to throw at it.

Hope this helps.

Answer 2

You may try List<T> and ToArray() method associate with List. And it's supported by XNA framework too ( MSDN ).

List is a successor to ArrayList and provide more features and strongly typed (A good comparison ).

About performance, List<T>.ToArray is a O(n) operation . And I suggest you to break your lengthy array to sort of portions which you can name with a key [Some sort of unique identifier to a region or so on] . And store relevant information in a List and use Dictionary like Dictionary<Key, List<T>> which could reduce operations involved. Also you can process required models with priority based approach which would give a performance gain over processing complete array at once.