Why is 199.96 - 0 = 200 in SQL?

Question

I have some clients getting weird bills. I was able to isolate the core problem:

SELECT 199.96 - (0.0 * FLOOR(CAST(1.0 AS DECIMAL(19, 4)) * CAST(199.96 AS DECIMA         


        

Answer 1

I need to start by unwrapping this a bit so I can see what's going on:

SELECT 199.96 - 
    (
        0.0 * 
        FLOOR(
            CAST(1.0 AS DECIMAL(19, 4)) * 
            CAST(199.96 AS DECIMAL(19, 4))
        )
    ) 

Now let's see exactly what types SQL Server is using for each side of the subtraction operation:

SELECT  SQL_VARIANT_PROPERTY (199.96     ,'BaseType'),
    SQL_VARIANT_PROPERTY (199.96     ,'Precision'),
    SQL_VARIANT_PROPERTY (199.96     ,'Scale')

SELECT  SQL_VARIANT_PROPERTY (0.0 * FLOOR(CAST(1.0 AS DECIMAL(19, 4)) * CAST(199.96 AS DECIMAL(19, 4)))  ,'BaseType'),
    SQL_VARIANT_PROPERTY (0.0 * FLOOR(CAST(1.0 AS DECIMAL(19, 4)) * CAST(199.96 AS DECIMAL(19, 4)))  ,'Precision'),
    SQL_VARIANT_PROPERTY (0.0 * FLOOR(CAST(1.0 AS DECIMAL(19, 4)) * CAST(199.96 AS DECIMAL(19, 4)))  ,'Scale')

Results:

numeric 5   2
numeric 38  1

So 199.96 is numeric(5,2) and the longer Floor(Cast(etc)) is numeric(38,1).

The rules for the resulting precision and scale of a subtraction operation (ie: e1 - e2) look like this:

Precision: max(s1, s2) + max(p1-s1, p2-s2) + 1
Scale: max(s1, s2)

That evaluates like this:

Precision: max(1,2) + max(38-1, 5-2) + 1 => 2 + 37 + 1 => 40
Scale: max(1,2) => 2

You can also use the rules link to figure out where the numeric(38,1) came from in the first place (hint: you multiplied two precision 19 values).

But:

• The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, it is reduced to 38, and the corresponding scale is reduced to try to prevent the integral part of a result from being truncated. In some cases such as multiplication or division, scale factor will not be reduced in order to keep decimal precision, although the overflow error can be raised.

Oops. The precision is 40. We have to reduce it, and since reducing precision should always cut off the least significant digits that means reducing scale, too. The final resulting type for the expression will be numeric(38,0), which for 199.96 rounds to 200.

You can probably fix this by moving and consolidating the CAST() operations from inside the large expression to one CAST() around the entire expression result. So this:

SELECT 199.96 - 
    (
        0.0 * 
        FLOOR(
            CAST(1.0 AS DECIMAL(19, 4)) * 
            CAST(199.96 AS DECIMAL(19, 4))
        )
    ) 

Becomes:

SELECT CAST( 199.96 - ( 0.0 * FLOOR(1.0 * 199.96) ) AS decimial(19,4))

I might even remove the outer cast, as well.

We learn here we should choose types to match the precision and scale we actually have right now, rather than the expected result. It doesn't make sense to just go for big precision numbers, because SQL Server will mutate those types during arithmetic operations to try to avoid overflows.

---

More Information:

• Precision, Scale, and Length using Sql_Variant_Property()
• Operator Precedence
• Data type precedence