How can I read lines with different number of data entries from text file into array in Matlab?

Question

I have a text file containing message and timestamps:

 {193220,message,ucontroller9,controller1,lowalarm,success},
 {193220,message,controller1,changer,{change,down},success},
 {193220,controlaction,tapchange_down,{oldsetting,0.975},{newsetting,0.95}},
 {193220,modelupdate_start,changeup,changer,193220},
 {14430,modelupdate_complete},
 {278480,message,ucontroller6,controller1,highalarm,success},
 {278480,message,controller1,changer,{change,up},success},
 ...

I would like to read these lines into an array and re-order the array based on the first column. My approach so far has been as follows:

fid = fopen('messagestore.txt')    % Open file
n=1    
while 1
string = fgetl(fid);               % get line
if ~ischar(string), break, end     % break if end of file
string = strrep(string,'[','');    %
string = strrep(string,']','');    %
string = strrep(string,'{','');    % strip out unwanted characters
string = strrep(string,'}','');    %
string = strrep(string,',',' ');   %
string = string(2:end);            % remove space at start

MessageArray(:,n) = textscan(string,'%f%s%s%s%s%s');  % format and save into array
n=n+1;
end
fclose(fid) 

This partially works but I am having a problem with discriminating between different message types (ie number of fields). I'm guessing that I could use the matlab sort function if I could just get the data into an array properly.

Is there a way to do this in matlab?

Answer 1

Here are some steps that should work:

0: Find an upper bounds for the number of fields in each line

1: Make sure you have nice rectangular data. Look at the number of commas present in each line minus the number of { brackets. (don't deduct the } or you will be counting doubles). This gives you the number of fields you have. Then add commas untill the maximum is achieved for each line.

2: Change the { and } into a standard text indicator like "

3: Import the result using xlsread() or similar.

Answer 2

Dennis is right in saying that you probably should do some preprocessing, since you have multiple sets of braces and such.

What you may be looking for is some way of basically sorting the lines of the file. If you're not worried about capturing the individual string fields, but just in re-ordering lines, you can read and sort according to the first field, which you know to be numbers. This is pretty hackish, but maybe you're looking for something like (this is essentially what Dennis suggests in the comments above):

clear all, clc
fid = fopen('messagestore.txt');
n=1;
nums = [];
while 1
    string = fgetl(fid);               % get line
    if ~ischar(string), break, end     % break if end of file

    for k = 1:length(string)
        if string(k) == ','
            break
        end
    end
    nums = [nums str2num(string(2:k-1))];
    MessageArray{n} = string;
    n=n+1;
end
[nums,I] = sort(nums);
MessageArray = MessageArray(I);
fclose(fid);

% Print some output
for k = 1:length(nums)
    disp(MessageArray{k})
end

which produces

{14430,modelupdate_complete},
{193220,message,ucontroller9,controller1,lowalarm,success},
{193220,message,controller1,changer,{change,down},success},
{193220,controlaction,tapchange_down,{oldsetting,0.975},{newsetting,0.95}},
{193220,modelupdate_start,changeup,changer,193220},
{278480,message,ucontroller6,controller1,highalarm,success},
{278480,message,controller1,changer,{change,up},success}

Now, I am not entirely sure how your timestamps work, here I literally sort the numerical value. But you should be able to get the basic idea from the code -- read the numbers from the first field into a vector and use that vector to sort the lines themselves. Whatever type of sorting your timestamps require can be performed by changing [nums,I] = sort(nums)