padding a list of torch tensors (or numpy arrays)

Question

Let's say I have a list as the following:

l = [torch.randn(2,3), torch.randn(2,4),torch.randn(2,5)]

I want to zero pad all of them in the second dimension, so they will extend as far as 5 elements (5 being the max number between the three of the elements in the second dimension). How can I do this. I tried this but failed:

from torch.nn.utils.rnn import pad_sequence
pad_sequence(l, batch_first=True, padding_value=0)

which caused the following error:

RuntimeError: The expanded size of the tensor (3) must match the existing size (4) at non-singleton dimension 1.  Target sizes: [2, 3].  Tensor sizes: [2, 4]

The equivalent answer in Numpy would also be appreciated.

Answer 1

One option is to use np.pad .

Example:

import numpy as np
a = np.random.randn(2, 3)
b = np.pad(a, [(0, 0), (0, 2)], mode='constant') 

Print a gives

[[ 1.22721163  1.23456672  0.51948003]
 [ 0.16545496  0.06609003 -0.32071653]]

Print b gives

[[ 1.22721163  1.23456672  0.51948003  0.          0.        ]
 [ 0.16545496  0.06609003 -0.32071653  0.          0.        ]]

The second argument of pad is pad_width which is a list of before/after paddings for each dimension. So in this example no padding in the first dimension and two paddings at the end of the second dimension.

There are lots of other mode options you can use so check out the docs.

For your particular problem you'll need to add an extra step to work out the padding for each array.

Edit

For pytorch I think you want torch.nn.functional.pad eg

import torch
t = torch.randn(2, 3)
torch.nn.functional.pad(t, (0, 2))

Edit 2

The torch.nn.utils.rnn.pad_sequence requires the trailing dimensions of all the tensors in the list to be the same so you need to some transposing for it to work nicely

import torch
# l = [torch.randn(2,3), torch.randn(2,4),torch.randn(2,5)]
# l = [i.transpose(0, 1) for i in l]  
# or simply make you tensors with switched dimensions
l = [torch.randn(3,2), torch.randn(4,2),torch.randn(5,2)]
out = torch.nn.utils.rnn.pad_sequence(l, batch_first=True)
# out will now be a tensor with shape (3, 5, 2)
# You can transpose it back to (3, 2, 5) with
out = out.transpose(1, 2)