pytorch RuntimeError: Expected object of scalar type Double but got scalar type Float

Question

I am trying to implement a custom dataset for my neural.network. But got this error when running the forward function. The code is as follows.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
import numpy as np

class ParamData(Dataset):
    def __init__(self,file_name):
        self.data = torch.Tensor(np.loadtxt(file_name,delimiter = ','))    #first place
    def __len__(self):
        return self.data.size()[0]
    def __getitem__(self,i):
        return self.data[i]

class Net(nn.Module):
    def __init__(self,in_size,out_size,layer_size=200):
        super(Net,self).__init__()
        self.layer = nn.Linear(in_size,layer_size)
        self.out_layer = nn.Linear(layer_size,out_size)

    def forward(self,x):
        x = F.relu(self.layer(x))
        x = self.out_layer(x)
        return x

datafile = 'data1.txt'

net = Net(100,1)
dataset = ParamData(datafile)
n_samples = len(dataset)

#dataset = torch.Tensor(dataset,dtype=torch.double)   #second place
#net.float()                                          #thrid place

net.forward(dataset[0])         #fourth place

In the file data1.txt is a csv formatted text file containing certain numbers, and each dataset[i] is a size 100 by 1 torch.Tensor object of dtype torch.float64 . The error message is as follows:

Traceback (most recent call last):
  File "Z:\Wrong.py", line 33, in <module>
    net.forward(dataset[0])
  File "Z:\Wrong.py", line 23, in forward
    x = F.relu(self.layer(x))
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\Python38\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "E:\Python38\lib\site-packages\torch\nn\functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm

It seems that I should change the dtype of the numbers in dataset to torch.double . I tried things like

• changing the line at the first place to self.data = torch.tensor(np.loadtxt(file_name,delimiter = ','),dtype=torch.double)

• changing the line at the fourth place to net.forward(dataset[0].double())

• uncommenting one of the two lines at the second or the thrid place

I think these are the solutions I have seen from similar questions, but they either give new errors or don't do anything. What should I do?

---

Update: So I got it working by changing the first place to

self.data = torch.from_numpy(np.loadtxt(file_name,delimiter = ',')).float()

which is weird because it is exactly the opposite of the error message. Is this a bug? I'd still like some explaining.

Answer 1

Now that I have more experience with pytorch, I think I can explain the error message. It seems that the line

RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm

is actually refering to the weights of the linear layer when the matrix multiplication is called. Since the input is double while the weights are float , it makes sense for the line

output = input.matmul(weight.t())

to expect the weights to be double .

Answer 2

In short : your data has type double but your model has type float , this is not allowed in pytorch because only data with the same dtype can be fed into the model.

In long : This issue is related to the default dtype of PyTorch and Numpy. I will first explain why this error happens and then suggest some solutions(but I think you will not need my solution once you understand the principle.)

• PyTorch has a couple of dtypes https://pytorch.org/docs/stable/tensors.html . Two of them are closely related to the issue you had:

1. torch.float32 (aka torch.float )
2. torch.float64 (aka torch.double )

• It's important to know the default dtype of PyTorch Tensors is torch.float32 (aka torch.float ). This means when you create a tensor, its default dtype is torch.float32 .try: torch.ones(1).dtype . This will print torch.float32 in default case. And also the model's parameters are of this dtype by default.

• In your case, net = Net(100,1) will create a model whose dtype of parameters are torch.float32

Then we need to talk about Numpy:

• The default dtype of Numpy ndarray is numpy.float64 . This means when you create a numpy array, its default dtype is numpy.float64 .try: np.ones(1).dtype . This will print dtype('float64') in default case.

• In your case, your data come from a local file loaded by np.loadtxt , so the data is first loaded as dtype('float64') (as a numpy array) and then converted to a torch tensor of dtype torch.float64 (aka torch.double ). This is what happens when you convert a numpy array to torch tensor: they will have the corresponding dtype.

I think now the issue is pretty clear, you have a model whose parameters are of torch.float32 (aka torch.float ) but tries to run it on data of torch.float64 (aka torch.double ). This is also what the error message tries to say: Expected object of scalar type Double but got scalar type Float for argument

Solutions:

1. You have alreay found one: convert your data to torch.float32 by calling tensor.float()
2. You can also specify the dtype when load the data: np.loadtxt(file_name,delimiter = ',',dtype="float32")

Answer 3

I came across this error while training a UNet in PyTorch. The training and test data was created from numpy arrays. Simply specifying the datatype while creating the tensors from numpy arrays resolved the error.

X_train, y_train = //read data files

train_x = torch.tensor(X_train, dtype=torch.float32) 

train_y = torch.tensor(y_train, dtype=torch.float32)