熵python实现

Question

I'm trying to rewrite this matlab/octave repo in python. I've came up on what seems to be an implementation of a function of entropy (see below). After some research,I've found on google that I can use scipy's entropy implementation for python. But after reading more on scipy's entropy formula (eg S = -sum(pk * log(pk), axis=0)), I'm having a doubt that those two are computing the same thing...

Could anyone confirm my thought please?

%�������� author by YangSong 2010.11.16 C230
%file:ys_sampEntropy.m
% code is called from line 101 of algotrading.m
%  =>   entropy180(i)=ys_sampEntropy(kmeans180s1(i,1:180));
% where kmeans180s1 is an array of size 100x181 containing the kmeans  
% centroids and the prize label at position 181.  

function sampEntropy=ys_sampEntropy(xdata)
m=2;
n=length(xdata);
r=0.2*std(xdata);%ƥ��ģ��������ֵ
%r=0.05;
cr=[];
gn=1;
gnmax=m;
while gn<=gnmax
      d=zeros(n-m+1,n-m);% ���ž��������ľ���  
      x2m=zeros(n-m+1,m);%���ű任�������      
      cr1=zeros(1,n-m+1);%���Ž����ľ���
      k=1;

      for i=1:n-m+1

          for j=1:m
              x2m(i,j)=xdata(i+j-1);
          end

      end
      x2m;

      for i=1:n-m+1

          for j=1:n-m+1

              if i~=j
                 d(i,k)=max(abs(x2m(i,:)-x2m(j,:)));%��������Ԫ�غ���ӦԪ�صľ���
                 k=k+1;
              end

          end

          k=1;
      end
      d;

      for i=1:n-m+1
          [k,l]=size(find(d(i,:)<r));%����RС�ĸ������͸�L
          cr1(1,i)=l;
      end
      cr1;

      cr1=(1/(n-m))*cr1;
      sum1=0;

      for i=1:n-m+1

          if cr1(i)~=0
             %sum1=sum1+log(cr1(i));
             sum1=sum1+cr1(i);
          end  %if����

      end  %for����

      cr1=1/(n-m+1)*sum1;
      cr(1,gn)=cr1;
      gn=gn+1;
      m=m+1;
end        %while����
cr;

sampEntropy=log(cr(1,1))-log(cr(1,2));

Answer 1

The code is pretty unreadable but it is still clear that this is not an implementation of the Shannon entropy calculation for discrete variables, as implemented in scipy. Instead this vaguely looks like the Kozachenko-Leonenko k-nearest neighbour estimator used to estimate the entropy of continuous variables (Kozachenko & Leonenko 1987).

The basic idea of that estimator is to look at the average distance between neighbouring data points. The intuition is that if that distance is large, the dispersion in your data is large and hence the entropy is large. In practice, instead of taking the nearest neighbour distance, one tends to take the k-nearest neighbour distance, which tends to make the estimate more robust.

The code shows some distance calculations

d(i,k)=max(abs(x2m(i,:)-x2m(j,:)));

and there is some counting of points that are nearer than some fixed distance:

[k,l]=size(find(d(i,:)<r));

However, it is also clear that this is not exactly the Kozachenko-Leonenko estimator but some butchered version of it.

If you do end up wanting to compute the Leonenko estimator, I have some code to that effect of my github:

https://github.com/paulbrodersen/entropy_estimators

EDIT:

After looking some more at this mess, I am no longer sure that he/she is not in fact using (attempting to use?) the classical Shannon information definition for discrete variables, even though the input is continuous:

  for i=1:n-m+1
      [k,l]=size(find(d(i,:)<r));%����RС�ĸ������͸�L
      cr1(1,i)=l;
  end
  cr1;

  cr1=(1/(n-m))*cr1;

The for loop counts the number of data points closer than r, and then the last line in the snippet divides that number by some interval to get a density.

Those densities are then summed below:

  for i=1:n-m+1

      if cr1(i)~=0
         %sum1=sum1+log(cr1(i));
         sum1=sum1+cr1(i);
      end  %if����

  end  %for����

But then we get these bits (again!):

  cr1=1/(n-m+1)*sum1;
  cr(1,gn)=cr1;

And

sampEntropy=log(cr(1,1))-log(cr(1,2));

My brain refuses to believe that the returned value could be your average log(p) but I am no longer 100% sure.

Either way, if you want to compute the entropy of a continuous variable, you should either fit a distribution to your data or you should use the Kozachenko-Leonenko estimator. And please write better code.

Answer 2

    ##Entropy
def entropy(Y):
    """
    Also known as Shanon Entropy
    Reference: https://en.wikipedia.org/wiki/Entropy_(information_theory)
    """
    unique, count = np.unique(Y, return_counts=True, axis=0)
    prob = count/len(Y)
    en = np.sum((-1)*prob*np.log2(prob))
    return en


#Joint Entropy
def jEntropy(Y,X):
    """
    H(Y;X)
    Reference: https://en.wikipedia.org/wiki/Joint_entropy
    """
    YX = np.c_[Y,X]
    return entropy(YX)

#Conditional Entropy
def cEntropy(Y, X):
    """
    conditional entropy = Joint Entropy - Entropy of X
    H(Y|X) = H(Y;X) - H(X)
    Reference: https://en.wikipedia.org/wiki/Conditional_entropy
    """
    return jEntropy(Y, X) - entropy(X)


#Information Gain
def gain(Y, X):
    """
    Information Gain, I(Y;X) = H(Y) - H(Y|X)
    Reference: https://en.wikipedia.org/wiki/Information_gain_in_decision_trees#Formal_definition
    """
    return entropy(Y) - cEntropy(Y,X)

Answer 3

This is what I use :

def entropy(data,bins=None):
    if bins is None : bins =  len(np.unique(data))
    cx = np.histogram(data, bins)[0]
    normalized = cx/float(np.sum(cx))
    normalized = normalized[np.nonzero(normalized)]
    h = -sum(normalized * np.log2(normalized))
    return h





"""
Approximate entropy : used to quantify the amount of regularity and the unpredictability of fluctuations 
over time-series data.

The presence of repetitive patterns of fluctuation in a time series renders it more predictable than a 
time series in which such patterns are absent. 
ApEn reflects the likelihood that similar patterns of observations will not be followed by additional 
similar observations.
[7] A time series containing many repetitive patterns has a relatively small ApEn, 
a less predictable process has a higher ApEn.

    U: time series
    The value of "m" represents the (window) length of compared run of data, and "r" specifies a filtering level.

    https://en.wikipedia.org/wiki/Approximate_entropy

Good m,r values :
   m = 2,3 : rolling window
   r = 10% - 25% of seq std-dev

"""

def approx_entropy(U, m, r):

    def _maxdist(x_i, x_j):
        return max([abs(ua - va) for ua, va in zip(x_i, x_j)])

    def _phi(m):
        x = [[U[j] for j in range(i, i + m - 1 + 1)] for i in range(N - m + 1)]
        C = [len([1 for x_j in x if _maxdist(x_i, x_j) <= r]) / (N - m + 1.0) for x_i in x]
        return (N - m + 1.0)**(-1) * sum(np.log(C))

    N = len(U)

    return abs(_phi(m + 1) - _phi(m))