Is floating point arithmetic stable? [duplicate]

Answer 1

In theory, on an IEEE 754 conforming system, the same operation with the same input values is supposed to produce the same result.

As Wikipedia summarizes it:

The IEEE 754-1985 allowed many variations in implementations (such as the encoding of some values and the detection of certain exceptions). IEEE 754-2008 has strengthened up many of these, but a few variations still remain (especially for binary formats). The reproducibility clause recommends that language standards should provide a means to write reproducible programs (i.e., programs that will produce the same result in all implementations of a language), and describes what needs to be done to achieve reproducible results.

As usual, however, theory is different from practice. Most programming languages in common use, C# included, do not strictly conform to IEEE 754, and do not necessarily provide a means to write a reproducible program.

Additionally, modern CPU/FPUs make it somewhat awkward to ensure strict IEEE 754 compliance. By default they will operate with "extended precision", storing values with more bits than a double internally. If you want strict semantics you need to pull values out of the FPU into a CPU register, check for and handle various FPU exceptions, and then push values back in -- between each FPU operation. Due to this awkwardness, strict conformance has a performance penalty, even at the hardware level. The C# standard chose a more "sloppy" requirement to avoid imposing a performance penalty on the more common case where small variations are not a problem.

None of this is often an issue in practice since most programmers have internalized the (incorrect, or at least misleading) idea that floating-point math is inexact. Additionally, the errors we're talking about here are all extremely small, enough so that they are dwarfed by the much more common loss of precision caused by converting from decimal.

Answer 2

No, it does not. The outcome of the calculation can differ per CPU, because the implementation of floating point arithmetic can differ per CPU manufacturer or CPU design. I even remember a bug in the floating point arithmetic in some Intel processors, which screwed up our calculations.

And then there is difference in how the code is evaluated bu the JIT compiler.

Answer 3

Jon's answer is of course correct. None of the answers however have said how you can ensure that floating point arithmetic is done in the amount of precision guaranteed by the specification and not more.

C# automatically truncates any float back to its canonical 32 or 64 bit representation under the following circumstances:

• You put in a redundant explicit cast: x + y might have x and y as higher-precision numbers that are then added. But (double)((double)x+(double)y) ensures that everything is truncated to 64 bit precision before and after the math happens.
• Any store to an instance field of a class, static field, array element, or dereferenced pointer always truncates. (Stores to locals, parameters and temporaries are not guaranteed to truncate; they can be enregistered. Fields of a struct might be on the short-term pool which can also be enregistered.)

These guarantees are not made by the language specification, but implementations should respect these rules. The Microsoft implementations of C# and the CLR do.

It is a pain to write the code to ensure that floating point arithmetic is predictable in C# but it can be done. Note that doing so will likely slow down your arithmetic.

Complaints about this awful situation should be addressed to Intel, not Microsoft; they're the ones who designed chips that make doing predictable arithmetic slower.

Also, note that this is a frequently asked question. You might consider closing this as a duplicate of:

Why differs floating-point precision in C# when separated by parantheses and when separated by statements?

Why does this floating-point calculation give different results on different machines?

Casting a result to float in method returning float changes result

(.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?

Coercing floating-point to be deterministic in .NET?

C# XNA Visual Studio: Difference between "release" and "debug" modes?

C# - Inconsistent math operation result on 32-bit and 64-bit

Rounding Error in C#: Different results on different PCs

Strange compiler behavior with float literals vs float variables

Answer 4

But what if the equation used to calculate the number is the same? can I assume the outcome would be the same too?

No, not necessarily.

In particular, in some situations the JIT is permitted to use a more accurate intermediate representation - e.g. 80 bits when your original data is 64 bits - whereas in other situations it won't. That can result in seeing different results when any of the following is true:

• You have slightly different code, e.g. using a local variable instead of a field, which can change whether the value is stored in a register or not. (That's one relatively obvious example; there are other much more subtle ones which can affect things, such as the existence of a try block in the method...)
• You are executing on a different processor (I used to observe differences between AMD and Intel; there can be differences between different CPUs from the same manufacturer too)
• You are executing with different optimization levels (e.g. under a debugger or not)

From the C# 5 specification section 4.1.6:

Floating-point operations may be performed with higher precision than the result type of the operation. For example, some hardware architectures support an "extended" or "long double" floating-point type with greater range and precision than the double type, and implicitly perform all floating-point operations using this higher precision type. Only at excessive cost in performance can such hardware architectures be made to perform floating-point operations with less precision, and rather than require an implementation to forfeit both performance and precision, C# allows a higher precision type to be used for all floating-point operations. Other than delivering more precise results, this rarely has any measurable effects. However, in expressions of the form x * y / z, where the multiplication produces a result that is outside the double range, but the subsequent division brings the temporary result back into the double range, the fact that the expression is evaluated in a higher range format may cause a finite result to be produced instead of an infinity.

Answer 5

In addition to the other answers, it's possible x/y != x/y even on the same machine.

Floating-point computations in x86 are done using 80-bit registers, but truncated to 64-bits when stored. So it's possible for the first division to be calculated, truncated, then loaded back into memory and compared against the non-truncated division.

See here for more information (it's a C++ link, but the reasoning is the same)

Answer 6

Frankly I wouldn't expect two places in the same codebase to return the same thing for x/y for the same x and y - in the same process on the same machine; it can depend on how exactly x and y get optimized by the compiler / JIT - if they are enregistered differently, they can have different intermediate precisions. A lot of operations are done using more bits than you expect (register size); and exactly when this gets forced down to 64 bits can affect the outcome. The choice of that "exactly when" can depends on what else is happening in the surrounding code.