C# decimal precision function

Question

I have a decimal function:

public decimal NGD(long TotalPages, long XHits, long YHits, long BothHits)
{
    return ((Math.Max(XHits, YHits)-BothHits)/(TotalPages - Math.Min(XHits, YHits)));
}

upon printing the arguments I get NGD(44000000000,36100000,5630000,3580000) which translates to return ((36100000 - 3580000 )/(44000000000 - 5630000)); yet in my application I get a 0 when it should be 0.00073918549

Answer 1

You're doing integer division, which can only return a truncated integer.

You need to cast one operand to a non-integral type (probably decimal ).

Answer 2

That's because you're using integer division, which truncates the result. Use return ((decimal)(36100000 - 3580000 )/((decimal)(44000000000 - 5630000))); for decimal division.

Answer 3

All of your arithmetic is done with long ( System.Int64 ) which is an integer type. Only after the division is finished (giving an integer result) will the quotient be converted to decimal .

Consider casting your dividend (or even just the leading term of it) to decimal ( System.Decimal ). Then all other long values will automatically be "promoted" to decimal along the way.

Of course you may instead change one parameter (or all parameters) to have type decimal . No explicit cast will be needed to convert an expression of type long into an method argument of type decimal .