Matlab：加速读取ascii文件

Question

I wrote this piece of code which works fine, but is way too slow for my purposes:

%%% load nodal data %%% 
path = sprintf('%sfile.dat',directory);
fid = fopen(path);

num_nodes = textscan(fid,'%s %s %s %s %d',1,'delimiter', ' ');
num_nodes = num_nodes{5};
header = textscan(fid,'%s',7,'delimiter', '\t');

k = 0;
while ~feof(fid)

    line        = fgetl(fid);
    [head,rem]  = strtok(line,[' ',char(9)]); 

    if head == '#'
        k = k+1;
        j = 1;
        time_steps(k)  = sscanf(rem, [' Output at t = %d']);        
    end

    if ~isempty(head)
        if head ~= '#'
            data(j,:,k)  = str2num([head rem]); 
            j = j+1;
        end
    end

end
fclose(fid);

nodal_data = struct('header',header,'num_nodes',num_nodes,'time_steps',time_steps,'data',data);

The ascii I am reading into Matlab looks something like this:

# Number of Nodes: 120453
#X                  Y                   Z                   depth               vel_x               vel_y               wse             
# Output at t = 0
       76456.003              184726             3815.75                   0                   0                   0             3815.75
       76636.003              184726             3728.25                   0                   0                   0             3728.25
       76816.003              184726                3627                   0                   0                   0                3627
       76996.003              184726             3527.75                   0                   0                   0             3527.75
       77176.003              184726              3371.5                   0                   0                   0              3371.5
# Output at t = 36000.788
       76456.003              184726             3815.75                   0                   0                   0             3815.75
       76636.003              184726             3728.25                   0                   0                   0             3728.25
       76816.003              184726                3627                   0                   0                   0                3627
       76996.003              184726             3527.75                   0                   0                   0             3527.75
       77176.003              184726              3371.5                   0                   0                   0              3371.5

While the code I wrote works for files very small, it blows up on me for larger ascii files. I already had to abort loading a ~25mb ascii (approximately 240k lines), which was just a test file. Later versions of the file will be ~500mb. Is there a way of speeding up the process of loading the file I am not happy with the 3 if-statements, but I did not know how to seperate '#' from numbers with a switch on head, especially because I was not able to distinguish 'head' by class, ie I was trying to check either for ischar or isnumeric, but as the variable 'head' is read as string, it will always be the case of ischar and never never isnumeric = true . I am also not very happy with using a tokenizer at all to being able to use the if-cases and then putting together the line here: str2num([head rem]); , as this probably consumes a lot of time. However, I did not know how else to do it. So if you have any useful suggestions as of how to adapt my code, I would highly appreciate them!

Have a good sunday and thank you in advance!

Answer 1

The code below does reads approx 70000 timesteps with 5 nodes per step in around 7 seconds. It does most of what your code does and it should be easy enough to add the extra features of your code. There will be other ways of doing this faster but hopefully this should be adequate.

filename = 'd:\temp\input.txt';

filetext = fileread(filename);
headerLines = 2;
valuesPerLine = 7;
expr = '[^\n]*[^\n]*';
lines = regexp(filetext, expr, 'match');
isTimeStep = cellfun(@(x) strncmp(x,'#',1), lines );
numTimeSteps = sum(isTimeStep)-headerLines;
nodesPerStep = ((length(lines)-headerLines) / numTimeSteps ) - 1;
data = zeros(nodesPerStep, valuesPerLine, numTimeSteps);

for timeStep = 1:numTimeSteps
    lineIndex = headerLines + (timeStep-1) * (nodesPerStep + 1) + 2;
    for node = 1:nodesPerStep
        data(node, :, timeStep ) = sscanf(lines{lineIndex},'%f');
        lineIndex = lineIndex + 1;
    end    
end

Just tried it on a 2 million line file (340000 time steps with 5 nodes per step) and it took approx 36 seconds to run.

If you want a solution that doesn't have coded loops, you could replace from the code from

data = zeros(....

with

values = cellfun(@(x) sscanf(x,'%f'),lines(~isTimeStep),'uniformoutput',false);
data = reshape(cell2mat(values), nodesPerStep, valuesPerLine, numTimeSteps);

but it takes about 50% longer to run.

Answer 2

First thing to do before you change anything is to PRE-ALLOCATE all output arrays:
Your code outputs time_steps and data , all growing inside the loop. This can kill you performance.

Assuming there are always five lines between each time step .

Add the following lines before the loop

data = reshape( NaN( num_nodes, 7 ), [], 7, 5 ); % assuming 7 columns and 5 lines for each time step
time_steps = NaN( num_nodes / 5 );

after the loop just discard remaining NaNs

data( isnan(data) ) = [];
time_step( isnan(time_step) ) = [];