Fanning Software Consulting

One Plus One Equals One

QUESTION: Wayne Landsman reported the following observation on the IDL newsgroup recently.

I had a program recently fail because I did not realize that adding 1 to a number does not necessarily change its value. ;-)

   IDL> a = 4611686018427387947LL    IDL> b = double(a)    IDL> help,a,b
   A               LONG64    =    4611686018427387947
   B               DOUBLE    =    4.6116860e+18    IDL> print,a EQ b       1
   IDL> print,a+1       4611686018427387948    IDL> print,(a+1) EQ b       1

So b is equal to both a and a+1. My guess is that the values are getting converted to double precision prior to the equality test. But the LONG64 variable has more precision than a double precision variable, and that precision is lost during the conversion.

I'm not sure that there a good general solution for comparing between different data types. But one needs to be careful when comparing LONG64 and double variables.

ANSWER: Mr Penteado responded with this insightful message.

I think that converting integer types to floating point types is the usual way languages deal with operations that mix them, so this is not an IDL specific issue. Probably because it happens more often that the floating number is not an integer, and the integer is small enough to be represented exactly in the floating type.

Note that long64(b) is not equal to a, because double types are not precise to 1 part in 19. Double precision is only good to about 15 digits. For a number of that size in a double, only additions of the order of 1000 would change the value of b.

For that number to fit in a floating type you would need a quadruple precision type (128 bits), which gets to 34 digits. But IDL does not currently have such a type.

Wayne concedes the point with this comment.

Ideally, I think one would want two numeric variables, a and b, to be considered equal if

   double(a) = double(b)   *and*   long64(a)  = long64(b) 

since the double variable is higher precision in allowing fractional values, but the long64 variable is higher precision in preserving all digits of very large integers. But I agree that the IDL approach is standard among languages that allow comparison of data of different types.

Version of IDL used to prepare this article: IDL 7.1.2

Web Coyote's Guide to IDL Programming