floating-point-precision

int main() { float x=3.4e2; printf("%f",x); return 0; } Output: 340.000000 // It's ok. But if write x=3.1234e2 the output is 312.339996 and if x=3.12345678e2 the output is 312.345673 . Why are the outputs like these? I think if I write x=3.1234e2 the output should be 312.340000 , but the actual output is 312.339996 using GCC compiler. Not all fractional numbers have an exact binary equivalent so it is rounded to the nearest value. Simplified example, if you have 3 bits for the fraction, you can have: 0 0.125 0.25 0.375 ... 0.5 has an exact representation, but 0.1 will be shown as 0.125. Of

XCode 6.3.1 Swift 1.2 let value: Int = 220904525 let intmax = Int.max let float = Float(value) // Here is an error probably let intFromFloat = Int(float) let double = Double(value) println("intmax=\(intmax) value=\(value) float=\(float) intFromFloat=\(intFromFloat) double=\(double)") // intmax=9223372036854775807 value=220904525 float=2.20905e+08 intFromFloat=220904528 double=220904525.0 The initial value is 220904525. But when I convert it to float it becomes 220904528. Why? This is due to the way the floating-point format works. A Float is a 32-bit floating-point number, stored in the IEEE

I have an XSD that requires me to use a BigDecimal for a lat/lon. I currently have the lat/lon as doubles, and convert them to BigDecimal, but I am only required to use about 12 places of precision. I have not been able to figure out how to set that. Can anyone help me with this? You can use setScale() e.g. double d = ... BigDecimal db = new BigDecimal(d).setScale(12, BigDecimal.ROUND_HALF_UP); The title of the question asks about precision. BigDecimal distinguishes between scale and precision. Scale is the number of decimal places. You can think of precision as the number of significant

I am getting a very strange value for my (1,1) entry for my BinvA matrix I am just trying to invert B matrix and do a (B^-1)A multiplication. I understand that when I do the calculation by hand my (1,1) is supposed to be 0 but instead I get 1.11022302e-16. How can I fix it? I know floating point numbers can't be represented to full accuracy but why is this giving me such an inaccurate response and not rounding to 0 is there any way I can make it more accurate? Her is my code: import numpy as np A = np.array([[2,2],[4,-1]],np.int) A = A.transpose() B = np.array([[1,3],[-1,-1]],np.int) B = B

As part of a unit test, I need to test some boundary conditions. One method accepts a System.Double argument. Is there a way to get the next-smallest double value? (i.e. decrement the mantissa by 1 unit-value)? I considered using Double.Epsilon but this is unreliable as it's only the smallest delta from zero, and so doesn't work for larger values (i.e. 9999999999 - Double.Epsilon == 9999999999 ). So what is the algorithm or code needed such that: NextSmallest(Double d) < d ...is always true. If your numbers are finite, you can use a couple of convenient methods in the BitConverter class: long

I'm looking through old exam questions (currently first year of uni.) and I'm wondering if someone could explain a bit more thoroughly why the following for loop does not end when it is supposed to. Why does this happen? I understand that it skips 100.0 because of a rounding-error or something, but why? for(double i = 0.0; i != 100; i = i +0.1){ System.out.println(i); } The number 0.1 cannot be exactly represented in binary, much like 1/3 cannot be exactly represented in decimal, as such you cannot guarantee that: 0.1+0.1+0.1+0.1+0.1+0.1+0.1+0.1+0.1+0.1==1 This is because in binary : 0.1=