Ruby float precision

Question

As I understand it, Ruby (1.9.2) floats have a precision of 15 decimal digits. Therefore, I would expect rounding float x to 15 decimal places would equal x

Answer 1

Part of the problem is that 0.33 does not have an exact representation in the underlying format, because it cannot be expressed by a series of 1 / 2ⁿ terms. So, when it is multiplied by 10 a number slightly different than 0.33 is being multiplied.

For that matter, 3.3 does not have an exact representation either.

Part One

When numbers don't have an exact base-10 representation, there will be a remainder when converting the least significant digit for which there was information in the mantissa. This remainder will propagate to the right, possibly forever, but it's largely meaningless. The apparent randomness of this error is due to the same reason that explains the apparently-inconsistent rounding you and Matchu noticed. That's in part two.

Part Two

And this information (the right-most bits) is not aligned neatly with the information conveyed by a single decimal digit, so the decimal digit will typically be somewhat smaller than its value would have been if the original precision had been greater.

This is why a conversion might round to 1 at 15 digits and 0.x at 16 digits: because a longer conversion has no value for the bits to the right of the end of the mantissa.

Answer 2

Well, though I'm not certain on the details of how Ruby handles floats internally, I do know why this particular bit of code is failing on my box:

 > x = (0.33 * 10)
=> 3.3000000000000003
 > x.round(15)
=> 3.300000000000001

The first float keeps 16 decimal places for a total of 17 digits, for whatever reason. So, rounding to 15 discards those digits.