结构数组导入-内存预分配问题

Question

I have a few .mat files, each of them has an array of structures (of unknown length) called DATA. I want to import all these structures in a single array, but I don't want to use this code:

FileNames = strcat('file',num2str((1:N)''),'.mat'); 
DATATemp = [];

for int = 1:length(FileNames)
    load(FileNames(int,:));
    DATATemp=[DATATemp DATA];
end 

DATA = DATATemp;

because it does not prealocate the memory for the array.

Are there any clever ways of doing that?

Answer 1

If the length is short enough that you can over-allocate memory, you can do something like this: Pick an array size that is way larger than what you will ever see, and then trim it back down after you are done.

FileNames = strcat('file',num2str((1:N)''),'.mat'); 
DATATemp = zeros(1e6,1);

idx = 1;
for int = 1:length(FileNames)
    load(FileNames(int,:));
    idx_end = idx + length(DATA) - 1;
    DATATemp(idx:idx_end) = DATA;
    idx = idx_end + 1;
end 

DATA = DATATemp(1:idx_end);

However, if you are talking about a LOT of data, or just want to cover all your bases, a more rigorous solution is to allocate in chunks

FileNames = strcat('file',num2str((1:N)''),'.mat');

CHUNK_SIZE = 1e6; 
DATATemp = zeros(INIT_SIZE,1);

idx = 1;
for int = 1:length(FileNames)
    load(FileNames(int,:));
    idx_end = idx + length(DATA) - 1;
    if idx_end > length(DATATemp)
        DATATemp = [DATATemp zeros(INIT_SIZE ,1);
    DATATemp(idx:idx_end) = DATA;
    idx = idx_end + 1;
end 

DATA = DATATemp(1:idx_end);

Just make sure that your CHUNK_SIZE is significantly larger than the size of a typical individual file. I picked 1e6 here. That's what I would pick if I was loading ~20 files with an average size of 1e5 each. That way, although I'm still concatenating more space, it's much less often. This may not be very clever, but I hope it helps.

Also note that if you are loading files from a network drive that will slow things down immensely.