Extracting values in a matrix based on values of another matrix without a for loop in Octave

Question

I'd like to know if there's a way to operate on the values of one matrix, based separatedly on the values of each row from another matrix, without using a for loop.

One specific example below.

data is a ~500k row matrix with three columns: the first is a list of hours/dates (I have them as serial numbers) covering 24 hours a day for several years, the second column is a location ID, and the third one is the electricity cost in that location.

hours has two columns: the first one is a list of dates/hours as a serial number and the second one is a certain upper limit specific for that hour.

What I need to do is find, for every date and hour in hours , the largest value of electrical cost that is below the upper limit set in hours , and save the ID of the corresponding location in a third column of hours if the id is unique, or zero if there's no location that satisfy the condition or if there's more than one.

A symplified version of the code I'm using:

#Add a third column for the hours matrix
ids=zeros(rows(hours),1)
hours=horzcat(hours,ids)


for i=1:rows(hour)
  #Get the data for all locations in that hour
  idx=(data(:,1)==hour(i,1) )   
  hourlydata=(data(idx,:))        

  #Get the ID for the maximum below the limit in that hour
  idx=(hourlydata(:,3)<hour(i,2))
  idlimit=(hourlydata(idx,3)==max(hourlydata(idx,3))   

  dataid=hourlydata(idlimit,2)

  #Check if the data exists and is unique
  if(rows(dataid)==1)
     id=dataid(1,1)
  else
     id=0
  endif  

  #Save the ID
  hour(i,3)=id
endfor

Is there any way to do this without using the for loop?

(I already optimized the data matrix to make it as small as possible, but it's still pretty big, so I might encounter memory constraints when trying to implement a solution)

Answer 1

You can work with arrayfun. Suppose your data is

data = [datenum('2014-01-17'), 1, 13;datenum('2014-01-18'), 2, 7]
hours = [datenum('2014-01-17'), 17; datenum('2014-01-18'), 3]

Then define a selector function

function id = select( d, limit, data )
  idx = (data(:,1)==d );

  hourlydata = (data(idx,:));

  idx = (hourlydata(:,3)<limit);
  idlimit = (hourlydata(idx,3)==max(hourlydata(idx,3)));   

  dataid=hourlydata(idlimit,2);

  if length(dataid)==1
     id=dataid(1,1)
  else
     id=0
  end

end

Now we find a vector the same size as hours containing the id

arrayfun(@(d, limit) select(d, limit, data), hours(:,1), hours(:,2))
ans =
 1
 0

You can easily merge this vector with hours.

Now i doubt that this is much faster, but no loop. Works with MATLAB, haven't checked with Octave.