(10*1.11=11.1) evaluates as FALSE. How to correct it

后端 未结 1 555
谎友^
谎友^ 2020-12-20 01:56

(10*1.11=11.1) evaluates as FALSE.

a = 10
b = 1.11
c = 11.1
\' (mathematically: 10*1.11=11.1)
debug.print a*b = c

I get False (incorrect) r

相关标签:
1条回答
  • 2020-12-20 02:33

    Floating point arithmetics is almost never correct, when .1 is involved. The reason is that .1 cannot be represented as a 1/2^n within 16 digits of n.

    These are a few numbers, with which there would not exist problems in floating point, because they can be represented as 1/2^n:

    • 0.125 = 1/2^3
    • 0.5 = 1/2^1
    • 0.375 = 1/2^2 + 1/2^3

    Thus, the type decimal comes to solve the problem. This is the best way:

    Sub TestMe()
    
        Dim a, b, c 'declaring as Variant, as Decimal cannot be declared in VBA
    
        a = CDec(10)
        b = CDec(1.11)
        c = CDec(11.1)
        Debug.Print a * b = c
    
    End Sub
    
    0 讨论(0)
提交回复
热议问题