difference between all values in csv columns - python

Question

At first, I'm sorry for my so poor level in python, so I have the next problem:

1) I red a lot of answers at this resource, but nothing work for me ( np.abs(a.values[:,np.newaxis]-a2.values) and simple np.diff() and a lot of another ways)

2!) I have csv file the following form:

 A  12 43 51 10 74
 B  14 32 31 27 23
 C  13 62 13 33 82
 D  18 31 73 70 42

and I need receive residual between all columns in raws, so

A:12-43 12-51 12-10 12-74... 43-12 43-51 43-10 43-74...
B:12-43 12-51 12-10 12-74... 43-12 43-51 43-10 43-74...

after that I need power 2 in 12-43 12-51 12-10 12-74... 43-12 43-51 43-10 43-74...

I understand, that pandas good works with tables, but how I can to make this?

And if you can, please in what way I need go, that to cut-off 10% of extreme results? Thank you very much for you attention and feature help.

Answer 1

I suggest using numpy . For computing the difference you can do

>>> a = numpy.array([[12, 43, 51, 10, 74],
...                  [14, 32, 31, 27, 23],
...                  [13, 62, 13, 33, 82],
...                  [18, 31, 73, 70, 42]])
>>> difference_matrix = numpy.repeat(a, a.shape[-1], axis=-1) - numpy.tile(a, a.shape[-1])
>>> difference_matrix
array([[  0, -31, -39,   2, -62,  31,   0,  -8,  33, -31,  39,   8,   0,
         41, -23,  -2, -33, -41,   0, -64,  62,  31,  23,  64,   0],
       [  0, -18, -17, -13,  -9,  18,   0,   1,   5,   9,  17,  -1,   0,
          4,   8,  13,  -5,  -4,   0,   4,   9,  -9,  -8,  -4,   0],
       [  0, -49,   0, -20, -69,  49,   0,  49,  29, -20,   0, -49,   0,
        -20, -69,  20, -29,  20,   0, -49,  69,  20,  69,  49,   0],
       [  0, -13, -55, -52, -24,  13,   0, -42, -39, -11,  55,  42,   0,
          3,  31,  52,  39,  -3,   0,  28,  24,  11, -31, -28,   0]])

If you want to square the result you can simply apply it to the matrix and each element will be squared:

>>> difference_matrix ** 2
array([[   0,  961, 1521,    4, 3844,  961,    0,   64, 1089,  961, 1521,
          64,    0, 1681,  529,    4, 1089, 1681,    0, 4096, 3844,  961,
         529, 4096,    0],
       [   0,  324,  289,  169,   81,  324,    0,    1,   25,   81,  289,
           1,    0,   16,   64,  169,   25,   16,    0,   16,   81,   81,
          64,   16,    0],
       [   0, 2401,    0,  400, 4761, 2401,    0, 2401,  841,  400,    0,
        2401,    0,  400, 4761,  400,  841,  400,    0, 2401, 4761,  400,
        4761, 2401,    0],
       [   0,  169, 3025, 2704,  576,  169,    0, 1764, 1521,  121, 3025,
        1764,    0,    9,  961, 2704, 1521,    9,    0,  784,  576,  121,
         961,  784,    0]])

Answer 2

pandas doesn't easily accept arrays as elements, so numpy is a good help here.

First make all the differences, by line ( axis=1 ) :

data="""
A 12 43 51 10 74
B 14 32 31 27 23
C 13 62 13 33 82
D 18 31 73 70 42
""" 
pd.read_table(io.StringIO(data),header=None,index_col=0,sep=' ')

all_differences=np.apply_along_axis(lambda x:np.subtract.outer(x,x).ravel(),axis=1,arr=df)

Then sort for cut-off :

all_differences.sort(axis=1)

and select the good values, discarding the 0 resulting from L[i]-L[i] .

n=df.shape[1]

cutoff =[i for i in range(n*n)  if  n*n*5//100<=i<n*(n-1)//2 or  n*(n+1)//2<=i<n*n*95//100]

res=2.**all_differences[:,cutoff]