floating-point-precision

问题 I have to put the price of some items inside a mysql table. When creating the table I'm using DECIMAL(10,2) as I don't need more than 2 digits after the comma (for example: 123,45 would be accepted as an input but 123,456 would be rounded to 123,45 with PHP). First question: using DECIMAL(10,2) , how do I know how many numbers may be stored before the comma? I know it is not 10, as 10 is just the precision Mysql uses when doing math with those numbers: so where the length of the number itself

问题 All is in the title: does IEEE-754 float , double and quad guarantee exact representation of -2 , -1 , -0 , 0 , 1 , 2 ? 回答1: It guarantees precise representations of all integers until the number of significant binary digits exceeds the range of the mantissa. 回答2: IEEE 754 floating point numbers can be used to store precisely integers of a certain ranges. For example: binary32 , implemented in C/C++ as float , provides 24 bits of precision and therefore can represent with full precision 16

问题 Okay so I get that some numbers can't be represented properly in binary just like 1/3 can't be fully represented in decimal. So how come when I console.log(0.3) it returns 0.3 but when I console.log(0.1 + 0.2) it returns the 0.30000000000000004 How come it is accounting for the error (if it even is) when simply outputting 0.3 but doesn't when the addition occurs? 回答1: Suppose we approximate 1/3 and 2/3 in decimal. 1/3 = 0.333 2/3 = 0.667 and we add 1/3+1/3: 1/3+1/3 = 0.333 + 0.333 = 0.666 We

问题 Okay so I get that some numbers can't be represented properly in binary just like 1/3 can't be fully represented in decimal. So how come when I console.log(0.3) it returns 0.3 but when I console.log(0.1 + 0.2) it returns the 0.30000000000000004 How come it is accounting for the error (if it even is) when simply outputting 0.3 but doesn't when the addition occurs? 回答1: Suppose we approximate 1/3 and 2/3 in decimal. 1/3 = 0.333 2/3 = 0.667 and we add 1/3+1/3: 1/3+1/3 = 0.333 + 0.333 = 0.666 We

问题 I'm sure the answer is obvious but I can't find it without looking at the source. What is gnuplot's internal representation of floating point numbers? Is it the platform's double ? Does it use its own internal representation? Can it do arbitrary precision? 回答1: A quick google search will turn up that calculations are done in double precision whenever possible, however, there's a little sublety going on here. The range of an IEEE double precision number should go up to more than 1.797e308 ,

问题 I have a file.txt with hundreds of numbers. They have many digits (max 20) after the point and I need to get them all without truncation, otherwise they introduce errors in the following computations. I made these numbers with matlab so it has a monstrous precision but now I must replicate this behaviour in my program. I've done this way: fstream in; in.open(file.txt, ios::in); long double number; in>>number; I also tried this in.precision(20); in>>number; before each ">>" operation but it is

问题 I'm trying to generate a string that involves an occasional float with trailing zeros. This is a MWE of the text string and my attempt at removing them with {0:g} : xn, cod = 'r', 'abc' ccl = [546.3500, 6785.35416] ect = [12.350, 13.643241] text = '${}_{{t}} = {0:g} \pm {0:g}\;{}$'.format(xn, ccl[0], ect[0], cod) print text Unfortunately this returns: ValueError: cannot switch from automatic field numbering to manual field specification This question Using .format() to format a list with

问题 I'm trying to generate a string that involves an occasional float with trailing zeros. This is a MWE of the text string and my attempt at removing them with {0:g} : xn, cod = 'r', 'abc' ccl = [546.3500, 6785.35416] ect = [12.350, 13.643241] text = '${}_{{t}} = {0:g} \pm {0:g}\;{}$'.format(xn, ccl[0], ect[0], cod) print text Unfortunately this returns: ValueError: cannot switch from automatic field numbering to manual field specification This question Using .format() to format a list with

问题 I'm working on a new version of my Mandelbrot screensaver and I'm running out of floating point accuracy - simple double values don't have enough significant figures for my needs. More significant figures = greater levels of zooming into the fractal Back when I wrote a version of this screensaver in Delphi 7, I used the extended floating point type, 80 bits in size. In .NET, I could switch to decimal, but the performance hit for this is terrible, slowing down fractal generation by a factor of

问题 I'm working on a new version of my Mandelbrot screensaver and I'm running out of floating point accuracy - simple double values don't have enough significant figures for my needs. More significant figures = greater levels of zooming into the fractal Back when I wrote a version of this screensaver in Delphi 7, I used the extended floating point type, 80 bits in size. In .NET, I could switch to decimal, but the performance hit for this is terrible, slowing down fractal generation by a factor of