SQL Server: Calculation with numeric literals

Question

I did some testing with floating point calculations to minimize the precision loss. I stumbled across a phenomen I want to show here and hopefully get an explanation.

Wh

Answer 1

I think it should be understood what is going on behind the scenes for future reference in similar cases.

Literal numerical values with decimal point excluding scientific notation represent Decimal data type which is stored as smallest possible Decimal type. Same quote as Lieven Keersmaekers's from: https://msdn.microsoft.com/en-us/library/ms191530%28SQL.90%29.aspx#_decimal

In Transact-SQL statements, a constant with a decimal point is automatically converted into a numeric data value, using the minimum precision and scale necessary. For example, the constant 12.345 is converted into a numeric value with a precision of 5 and a scale of 3.

The trailing zeros on the right of decimal point specify scale. The leading zeros left of decimal point are ignored.

Some examples:

1.0  -> Decimal(2,1)
60.0 -> Decimal(3,1)
1.00 -> Decimal(3,2)
01.0 -> Decimal (2,1)

Another point to consider is Data Type precedence. When an operator combines two expressions of different data types, the rules for data type precedence specify that the data type with the lower precedence is converted to the data type with the higher precedence. And yet another point to consider is if we do arithmetic operations on Decimal types that the resulting Decimal type, i.e. precision and scale depend on both operands and operation itself. This is described in document Precision, Scale, and Length.

So, part of your expression in parenthesis

( 1.0 / 60.0 ) is evaluated to 0.016666 and the resulting type is Decimal (8,6)

using above rules about Precision and scale of Decimal expressions. In addition the banker's rounding or rounding to even is used. It is important to note different rounding for Decimal and float type are used. If we continue the expression

1.0 / 0.016666 is evaluated to 60.002400096 and the resulting type is Decimal (17,10)

So the part of the discrepancy is due to different rounding being used for decimal types than for float.

In accordance to the above rules it would be sufficient to use just one cast inside parenthesis. Every other literal will be promoted to float in accordance with Data Type Precedence rules.

1.0 / (1.0 / cast(60.0 as float))

And one more IMPORTANT thing. Even this float expression does not calculate exact result. It is just so that the front end (SSMS or whatever) rounds the value to (I guess) precision 6 digits and then truncates trailing zeros. So i.e. 1.000001 becomes 1.

Simple, isn't it?