rounding-error

C# Rounding MidpointRounding.ToEven vs MidpointRounding.AwayFromZero

帅比萌擦擦* 提交于 2019-11-29 17:04:10
问题 In C# Is there any difference in the accuracy of the two decimal rounding strategies MidpointRounding.ToEven and MidpointRounding.AwayFromZero ? I mean do both ensure an even distribution amongst the numbers that are rounded to, or is one rounding strategy over representing the rounded numbers compared to the other? 回答1: From MSDN: By default, Math.Round uses MidpointRounding.ToEven. Most people are not familiar with "rounding to even" as the alternative, "rounding away from zero" is more

Rounding Inaccuracies When Combining Areas in Java?

江枫思渺然 提交于 2019-11-29 14:40:31
I'm working with Areas in Java. My test program draws three random triangles and combines them to form one or more polygons. After the Areas are .add() ed together, I use PathIterator to trace the edges. Sometimes, however, the Area objects will not combine as they should... and as you can see in the last image I posted, extra edges will be drawn. I think the problem is caused by rounding inaccuracies in Java's Area class (when I debug the test program, the Area shows the gaps before the PathIterator is used), but I don't think Java provides any other way to combine shapes. Any solutions?

Rounding differences on Windows vs Unix based system in sprintf

半城伤御伤魂 提交于 2019-11-29 09:46:24
I have problem on UNIX based systems sprintf does not round up properly value. For example double tmp = 88888888888885.875 char out[512]; Thats 88,888,888,888,885.875 just to be easier on eyes. I am giving such specific and big example because it seems it works fine on smaller numbers. I am trying to use it in following way sprintf(out, "%021.2f", tmp); printf("out = %s\n", tmp); On windows this results in: out = 000088888888888885.88 On for example AIX, but shows in Linux as well: out = 000088888888888885.87 Why is this happening? Any ideas and how to make it behave same way on Win/Unix

Entity Framework Code First truncating my decimals

五迷三道 提交于 2019-11-29 09:25:23
I am using Entity Framework 6.x using the Code First approach on an MVC 5 application. In this particular situation my model (among other things) contains two properties named Latitude and Longitude: [Required, Range(-90, +90)] public decimal Latitude { get; set; } [Required, Range(-180, +180)] public decimal Longitude { get; set; } And when I performed the migration I got something like this CreateTable("ResProperty"), c => new { : Latitude = c.Decimal(nullable: false, precision: 10, scale: 8), Longitude = c.Decimal(nullable: false, precision: 11, scale: 8), : }) ... other stuff so both

Error subtracting floating point numbers when passing through 0.0

流过昼夜 提交于 2019-11-29 08:41:35
The following program: #include <stdio.h> int main() { double val = 1.0; int i; for (i = 0; i < 10; i++) { val -= 0.2; printf("%g %s\n", val, (val == 0.0 ? "zero" : "non-zero")); } return 0; } Produces this output: 0.8 non-zero 0.6 non-zero 0.4 non-zero 0.2 non-zero 5.55112e-17 non-zero -0.2 non-zero -0.4 non-zero -0.6 non-zero -0.8 non-zero -1 non-zero Can anyone tell me what is causing the error when subtracting 0.2 from 0.2? Is this a rounding error or something else? Most importantly, how do I avoid this error? EDIT: It looks like the conclusion is to not worry about it, given 5.55112e-17

R issue with rounding milliseconds

生来就可爱ヽ(ⅴ<●) 提交于 2019-11-29 07:13:21
Given the following issue with rounding milliseconds under R. How do I get around it so that the times are correct? > options(digits.secs=3) > as.POSIXlt("13:29:56.061", format='%H:%M:%OS', tz='UTC') [1] "2012-06-07 13:29:56.060 UTC" > as.POSIXlt("13:29:56.062", format='%H:%M:%OS', tz='UTC') [1] "2012-06-07 13:29:56.061 UTC" > as.POSIXlt("13:29:56.063", format='%H:%M:%OS', tz='UTC') [1] "2012-06-07 13:29:56.063 UTC" I noticed that this URL provides background information but doesn't solve my issue: Milliseconds puzzle when calling strptime in R . Also this URL touches on the issue but doesn't

Why does 99.99 / 100 = 0.9998999999999999 [duplicate]

房东的猫 提交于 2019-11-28 12:45:17
问题 This question already has answers here : Closed 9 years ago . Possible Duplicate: Dealing with accuracy problems in floating-point numbers Whereas 99.99 * 0.01 = 0.99 Clearly this is the age old floating point rounding issue, however the rounding error in this case seems quite large to me; what I mean is I might have expected a result of 0.99990000001 or some similar 'close' result. And for the record I got the same answer in a JavaVM and in a .Net environment. 回答1: Why 0.9998999999999999 is

Rounding Inaccuracies When Combining Areas in Java?

假如想象 提交于 2019-11-28 08:52:51
问题 I'm working with Areas in Java. My test program draws three random triangles and combines them to form one or more polygons. After the Areas are .add() ed together, I use PathIterator to trace the edges. Sometimes, however, the Area objects will not combine as they should... and as you can see in the last image I posted, extra edges will be drawn. I think the problem is caused by rounding inaccuracies in Java's Area class (when I debug the test program, the Area shows the gaps before the

Rounding differences on Windows vs Unix based system in sprintf

…衆ロ難τιáo~ 提交于 2019-11-28 03:08:57
问题 I have problem on UNIX based systems sprintf does not round up properly value. For example double tmp = 88888888888885.875 char out[512]; Thats 88,888,888,888,885.875 just to be easier on eyes. I am giving such specific and big example because it seems it works fine on smaller numbers. I am trying to use it in following way sprintf(out, "%021.2f", tmp); printf("out = %s\n", tmp); On windows this results in: out = 000088888888888885.88 On for example AIX, but shows in Linux as well: out =

What class to use for money representation?

China☆狼群 提交于 2019-11-27 23:29:17
What class should I use for representation of money to avoid most rounding errors? Should I use Decimal , or a simple built-in number ? Is there any existing Money class with support for currency conversion that I could use? Any pitfalls that I should avoid? I assume that you talking about Python. http://code.google.com/p/python-money/ "Primitives for working with money and currencies in Python" - the title is self explanatory :) Never use a floating point number to represent money. Floating numbers do not represent numbers in decimal notation accurately. You would end with a nightmare of