Storing 10,000 lines of data in arrays in VB.net

Question

I am a complete noob to VB.Net , and have had my share of growing pains. I'm starting to get a handle on what I need to do, though.

The program I am writing needs to take about 500 .csv files, sucks out the info from them line by line, stores the data into about four different arrays, then export the data into one long index.

Each line in the files starts with a code word, and contains between 5 and 20 fields of data. The code word determines how many fields there are, and how the data needs to be stored. If it's Code A, it needs to go into Array A . If it's Index B, it needs to go in Array B and set some variables for Arrays A, B, C, and D. Code C means it goes into Array C . And so on.

My problem is that I will not know how many lines of data there will be, so using a number of standard arrays. I've got the code figured out so that I have each line of data channeled into the the correct sub. But I am unsure how to STORE the data.
I will need to manipulate/sort the data in Array C, but will be able to just dump data into and suck it out of Index A, B, and D.

Should I use 2D arrays for all the indexes? Would collections work better? If so, which kind of collection would work better?

//Array A= 4 columns per row, unknown number (500) of rows

//Array B= 18 columns of columns, unknown number (10,000+) rows

//Array C= 3 columns, unknown number (2000) of rows, must be able to sort and alter

//Array D= 3 columns, unknown number (1000) rows.  

Thank you

Answer 1

In a nutshell:

Should I use 2D arrays for all the indexes?

no.

Would collections work better?

Yes, much better.

If so, which kind of collection would work better?

Generic lists ( List(Of T) ), where you define objects (classes) with fields the match the columns for each type of record in your csv data, and use those classes as the types for your lists.

For ArrayB, beware the Large Object Heap causing problems with OutOfMemoryExceptions. You may need to keep that mostly on disk.