Storing data from variable length xml files without classes or giant arrays Java

Question

I'm working on a java project with 4000+ xml files. The basic layout is:

<Codes>
     <Code>
          <Nums>
               <Num>56</Num>
               <Num>21</Num>
               <Num>4</Num>
          </Nums>
     </Code>
</Codes>

These xmls files can have any amount of "num"s, even up to 100,000+. I know that layout of the xml files are weird but I am unable to change the layout because more could be added later.

I need to import these numbers (grouped by xml file). I have already created methods for parsing. The way of storing them is what is troubling me. I could great a GIANT int array:

int[][] nums = new int[4000][100000]

But this would be extremely memory consuming because some of the xml files have as few as 15 num entries (it raised the mem consumption of the java process by 1gb). The other option I came up with was having an instanced class for every xml file. For numerous reasons I would be unable to do this.

These nums represent to pixel colors that make of photos. The data will not need to be modified, just read by other classes.

Is there another way around this?

Thanks for the help!

Answer 1

I'm not entirely sure what your doing but you should probably seriously consider using some persistence technology like a database.

You seem to be concerned with memory as you should because loading a bunch of files in memory (4000 with 100k lines of XML) is probably going to waste a lot of memory while also being slow to access. A massive integer array can be fast but you will probably have to load the entire array leading to very slow startup time.

With a durable database you load the xml files once and have fast access after that.