Apparent F#/BCL floating point bug

Question

The following is in FSI:

> System.Math.Round(0.2916, 2);;
val it : float = 0.29
> it * 100.;;
val it : float = 29.0
> int it;;
val it : int = 28
         


        

Answer 1

This is just how floating point numbers work. The number that appears as 29 in the output is actually slightly smaller than 29 (because floating point numbers are not precise):

> (System.Math.Round(0.2916, 2) * 100.0) - 29.0;;
val it : float = -3.552713679e-15

Answer 2

float is binary floating point data type. Not all decimal floating point values can be represented exactly as binary floating point. What FSI shows as 29 is actually 28.999999999999996.

If you want to store precise decimal floating point values, use decimal data type (to make a number literal decimal instead of float add m to it - e.g. 0.2916m).

> System.Math.Round(0.2916m, 2);;
val it : decimal = 0.29M
> it * 100m;;
val it : decimal = 29.00M
> int it;;
val it : int = 29

Answer 3

You can change the F# FSI Pretty Printer like Matlab "format long"

fsi.AddPrinter( fun (x:float) -> sprintf "%26.16e" x);;

FSI output

System.Math.Round( 0.2916, 2)   //  2.8999999999999998e-001

1./3.                           // 3.3333333333333331e-001