How can I implement locally connected layer in pure Numpy

Question

I would like to build a locally connected weight matrix that represents a locally connected neural network in pure python/numpy without deep learning frameworks like Torch or TensorFlow.

The weight matrix is a non-square 2D matrix with the dimension (number_input, number_output). (an autoencoder in my case; input>hidden)

So the function I would like to build, take the matrix dimension and the size of the receptive field (number of local connection) and give the associated weight matrix. I've already create a function like this, but for an input size of 8 and an output size of 4 (and RF = 4) my function output :

[[ 0.91822845 0.          0.          0.        ]
[-0.24264655 -0.54754138  0.          0.        ]
[ 0.55617366  0.12832513 -0.28733965  0.        ]
[ 0.27993286 -0.33150324  0.06994107  0.61184121]
[ 0.          0.04286912 -0.20974503 -0.37633903]
[ 0.          0.         -0.10386762  0.33553009]
[ 0.          0.          0.          0.09562682]
[ 0.          0.          0.          0.        ]]

but I would like :

[[ 0.91822845 0.          0.          0.        ]
[-0.24264655 -0.54754138  0.          0.        ]
[ 0.55617366  0.12832513  0.          0.        ]
[ 0          -0.33150324  0.06994107  0         ]
[ 0.          0.04286912 -0.20974503  0.        ]
[ 0.          0.         -0.10386762  0.33553009]
[ 0.          0.          0.11581854  0.09562682]
[ 0.          0.          0.          0.03448418]]

Here's my python code :

import numpy as np

def local_weight(input_size, output_size, RF):
    input_range = 1.0 / input_size ** (1/2)
    w = np.zeros((input_size, output_size))
    for i in range(0, RF):
        for j in range(0, output_size):
            w[j+i, j] = np.random.normal(loc=0, scale=input_range, size=1)
    return w

print(local_weight(8, 4, 4))

I look forward for your response!

Answer 1

The trick is in a small pad to work more comfortably (or control the limits).

Then you must define the step you will take with respect to the input (it is not more than the input / output). Once this is done you just have to fill in the gaps and then remove the pad.

import math
import numpy as np
def local_weight(input_size, output_size, RF):
    input_range = 1.0 / input_size ** (1/2)
    padding = ((RF - 1) // 2)
    w = np.zeros(shape=(input_size + 2*padding, output_size))
    step = float(w.shape[0] - RF) / (output_size - 1)
    for i in range(output_size):
        j = int(math.ceil(i * step))
        j_next = j + RF
        w[j:j_next, i] = np.random.normal(loc=0, scale=input_range, size=(j_next - j))
    return w[padding:-padding, :]

I hope that is what you are looking for.

EDIT: I think the implementation was misguided. I reimplement the function, we go by parts.

1. I calculate the radius of the receptive field (padding).

2. Determine the size of the W.

3. I calculate the step by removing the padding area so that I always stay inside.

4. I calculate the weights.

5. Remove the padding.