浮点精度因 pandas dataframe 中的元素而异

Question

I am trying to read a dataframe from a csv, do some calculations with it and then export the results to another csv. While doing that I noticed that the value 8.1e-202 is getting changed to 8.1000000000000005e-202 . But all the other numbers are represented correctly.

Example:

A example.csv looks like this:

id,e-value
ID1,1e-20
ID2,8.1e-202
ID3,9.24e-203

If I do:

import pandas as pd
df = pd.read_csv("example.csv")
df.iloc[1]["e-value"]
>>> 8.1000000000000005e-202

df.iloc[2]["e-value"]
>>> 9.24e-203

Why is 8.1e-202 being altered but 9.24e-203 isn't?

I tried to change the datatype that pandas is using from the default

df["e-value"].dtype
>>> dtype('float64')

to numpy datatypes like this:

import numpy as np
df = pd.read_csv("./temp/test", dtype={"e-value" : np.longdouble})

but this will just result in:

df.iloc[1]["e-value"]
>>> 8.100000000000000522e-202

Can someone explain to me why this is happening? I can't replicate this problem with any other number. Everything bigger or smaller than 8.1e-202 seems to work normally.

Answer 1

Float number representation is something not so simple. Not every real number can be represented and almost all (relatively speaking) are actually approximations. Is not like integers, the precision varies and python has a precision undefined float really.

Each floating point standar will have their own set of real numbers that can represent exactly. There's no work around.

https://en.wikipedia.org/wiki/Single-precision_floating-point_format https://en.wikipedia.org/wiki/IEEE_754-2008_revision

If the problem really is the arithmetic or comparison, you should consider if error will grow or decrease. For example multiplying by large numbers can grow the representation error.

And also, when comparing you should do things like math.is_close . Basically comparing the distance between the numbers.

If you are trying to represent and operate real numbers, that aren't irrational numbers. Like integers, fractions or decimal numbers with finite digits, you can also consider cast to the proper digit representation like: int, decimal or fraction.

See this for further ideas: https://davidamos.dev/the-right-way-to-compare-floats-in-python/#:~:text=How%20To%20Compare%20Floats%20in%20Python&text=If%20abs(a%20%2D%20b),rel_tol%20keyword%20argument%20of%20math .