Loss of precision when transferring data from hdf5 dataset to numpy array

Question

I am trying to copy data from a hdf5 dataset (f_one, in the below screenshot) into a numpy array, but am finding that I'm losing some precision.

The last line of the screenshot (the last print statement) should read

subid2[0] == subid 2 [1] .

I just accidentally deleted the 2 before taking the screenshot. The output is correct.

As you can see, Python seems to think these two numbers are exactly the same - however, I need the precision to distinguish between these two numbers when they are contained within a numpy array. Does anyone know how I can get this precision? In short, how can I get that last print statement to yield False?

By the way, the following:

    f_one['SubhaloID'][0] == f_one['SubhaloID'][1]

yields False . So there is some precision being lost when copying into the numpy array.

Answer 1

The problem is that your inputs are of integer type, but when creating a numpy array without specifying the data type, you ended up casting them as floats by default. To avoid this, use dtype=np.int64 when creating an numpy array for this data. Another option is to convert the existing integer array directly, so the type of its entries is inherited.

Here is a simplified example.

import numpy as np
a = [30000000200000000, 30000000200000001]
print(a[0]==a[1])           # False 
b = np.array(a)
print(b[0]==b[1])           # False, direct conversion still has integers 
c = np.array([])
for i in range(2):
    c = np.append(c, a[i])
print(c[0]==c[1])           # True, the entries are now floats
d = np.array([], dtype=np.int64)
for i in range(2):
    d = np.append(d, a[i])
print(d[0]==d[1])           # False, the entries were declared as integers

Check the types using type(c[0]) and type(d[0]) to see the difference.