Precision, Rounding, and data types in XSLT2.0

Question

This is a question about precision, rounding, data types, and using the xs:integer() function in XSLT 2.0. We have the following line of code:

<xsl:value-of select="xs:integer(round(InvoiceAmount * 100))"/>

When the value of InvoiceAmount in the source XML is -62.84, the result of the value-of is -6283 which is incorrect. We have found ways of working around this, but we really need to know what is going on here.

Thinking that it is an error with binary multiplication where the value is off by a small fraction, we've tried the following using format-number down to 32 decimal places, but that didn't show anything (returned -6284.0 with 0s to 32 places):

<xsl:value-of select="format-number(InvoiceAmount * 100,'0.00000000000000000000000000000000')"/>

Also, if we replace InvoiceAmount in the XSL by hard-coded -62.84 it gives the correct result (-6284). And if we don't cast to integer using xs:integer() it also gives the correct result. If we use xs:decimal() instead of xs:integer() it works. And if we convert the XML value to decimal immediately (see the following code) it works:

<xsl:value-of select="xs:integer(round(xs:decimal(InvoiceAmount) * 100))"/>

I do not know what data type it uses by default, but once it understands that InvoiceAmount is a decimal it is fine. So, we have ways of working around the issue, but we would like to know what it assumes is the data type of InvoiceAmount and then why multiplying by 100, rounding, and converting to an integer causes a problem.

Thanks,

Steve

Answer 1

Assuming there is no schema involved, the typed value of the element InvoiceAmount is xs:untypedAtomic - which essentially means a string, but one that adapts itself to the context in which it is used. If you do an explicit conversion to xs:decimal, the untypedAtomic value "-62.84" will convert exactly. If you don't do an explicit conversion, then when you multiply by 100, the value will be treated as an xs:double, and there is no xs:double value that is EXACTLY -62.84, so it has to find the nearest double.

The XSLT 2.0 rules for conversion from xs:untypedAtomic to xs:string can be found here:

http://www.w3.org/TR/xpath-functions/#casting-from-strings

which in turn defers to the rules in XML Schema 1.0 Part 2, which can be found here:

http://www.w3.org/TR/xmlschema-2/#double

and the relevant rule is

"A literal in the ·lexical space· representing a decimal number d maps to the normalized value in the ·value space· of double that is closest to d; if d is exactly halfway between two such values then the even value is chosen. This is the best approximation of d ([Clinger, WD (1990)], [Gay, DM (1990)]), which is more accurate than the mapping required by [IEEE 754-1985]."

Now, this creates a bit of a problem for implementations. The algorithms given by Clinger and Gay for finding the closest xs:double value are seriously complex, and many products have probably taken a short-cut by calling whatever their handy programming language library offers for string-to-double conversion, which is probably faster but less accurate.

XSD 1.1 admitted defeat on this one: "Since IEEE allows some variation in rounding of values, processors conforming to this specification may exhibit some variation in their ·lexical mappings·." But that post-dates XSLT 2.0, so a strictly conformant XSLT 2.0 processor must follow Clinger/Gay.

Saxon implements string-to-double conversion by (a) checking for some of the things that Java allows and XPath doesn't (eg hex digits), and then (b) calling Double.parseDouble(). The effect of this, when applied to the string "-62.84" is to produce a double whose precise numeric value is -62.840000000000003410605131648480892181396484375. Is this the closest double to -62.84? I've got some tools somewhere to check this, but I don't think they're on the machine I'm using today.

Multiplying this by 100 in double arithmetic gives a value that is EXACTLY -6284.

Since round() rounds to the nearest integer, and we have a value that is either exactly equal to an integer or at least very close, I fail to see how any conformant implementation can produce -6283. It's not as if the value is close to the midpoint between two integers, where some variation would be permissible under the more permissive XSD 1.1 rules.