integer-overflow

Sleeping in VBA (Integer Overflow!!)

有些话、适合烂在心里 提交于 2019-12-11 16:23:53
问题 In VBA you can Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long) to provide yourself with a sleep routine. However, the Long that must be passed to the routine appears to overflow for values in excess of 32000 milliseconds. Is there a way to sleep for longer periods of time without the complexity of stringing together several consecutive calls to the sleep routine? 回答1: No, it doesn't overflow, unless your code that calculates required number of milliseconds causes an overflow.

Python prevent overflow errors while handling large floating point numbers and integers

情到浓时终转凉″ 提交于 2019-12-11 07:37:39
问题 I am working on a python program to calculate numbers in the Fibonacci sequence . Here is my code: import math def F(n): return ((1+math.sqrt(5))**n-(1-math.sqrt(5))**n)/(2**n*math.sqrt(5)) def fib(n): for i in range(n): print F(i) My code uses this formula for finding the Nth number in the Fibonacci sequence: This can calculate many of the the numbers in the Fibonacci sequence but I do get overflow errors. How can I improve this code and prevent overflow errors? Note: I am using python 2.7.

Integer overflow with subtraction

僤鯓⒐⒋嵵緔 提交于 2019-12-11 06:23:53
问题 Okay so I need to implement a method that returns zero or one depending on if x-y results in an overflow. Example: subTract(0x80000000,0x80000000) = 1, subTract(0x80000000,0x70000000) = 0, I'm NOT looking for an implementation of this method. I don't understand which one supposedly results in an overflow, and that making it nearly impossible to start. Why does one of these cause an overflow? What defines an overflow with subtraction. assume the system uses 2's complement and a 32-bit

Multiplication overflow in C

感情迁移 提交于 2019-12-11 04:26:00
问题 I'm doing some security CTF practice and have this problem which I am stuck on. This is the C source code of the compiled program: int main(int i, long **a) { if(*a[1] * 0x1064deadbeef4601u == 0xd1038d2e07b42569u) //exec shell return 0; } Things confusing me: long** main argument won't compile when I turn off gcc flags so I can't reproduce problem on my computer. Is this a different compiler being used? Compiled program runs fine on CTF server. program repeatedly overflows during

Is INT_MIN subtracted from any integer considered undefined behavior?

倖福魔咒の 提交于 2019-12-11 03:16:12
问题 What if I have something like this: int a = 20; int min = INT_MIN; if(-a - min) //do something Assume that INT_MIN if positive is more than INT_MAX. Would min ever be converted by the compiler to something like -min as in -INT_MIN, which could be undefined? 回答1: You are right that unary minus applied to INT_MIN can be undefined, but this does not happen in your example. -a - min is parsed as (-a) - min . Variable min is only involved in binary subtraction, and the first operand only needs to

Precision loss when solving nonlinear equations with long integer parameters by mpreal.h

这一生的挚爱 提交于 2019-12-11 02:31:32
问题 I have a numerical computation problem which requires solving nonlinear equations (with long integers) in multiple precision. I tried an MPFR C++ wrapper from this link by Pavel: mpfr C++ wrapper by Pavel The wrapper can be downloaded here: mpfrc++-3.5.6.zip However, there is precision loss in the solution when handling very long integers (equations with small integers worked well). I tried three options as in the sample code below: to use the code immediately does not work with "constant

Add with carry on Word8

给你一囗甜甜゛ 提交于 2019-12-10 20:30:03
问题 I can't find a function addWithCarry :: Word8 -> Word8 -> (Word8, Bool) already defined in base . The only function documented as caring about carries seems to be addIntC# in GHC.Prim but it seems to never be pushed upwards through the various abstraction layers. I could obviously roll out my own by testing whether the output value is in range and it's in fact what I am currently doing but I'd rather reuse an (potentially more efficient) already defined one. Is there such a thing? 回答1: If you

java arithmetic

[亡魂溺海] 提交于 2019-12-10 20:29:33
问题 why this code returns wrong value? int i=Integer.MAX_VALUE+1; long l=Integer.MAX_VALUE+1; System.out.println(l); System.out.println(i); 回答1: When you add 1 to Integer.MAX_VALUE it overflows and wraps around to Integer.MIN_VALUE . This happens because Java uses two's complement to represent integers. An example in 4 bits: 0000 : 0 0001 : 1 ... 0111 : 7 (max value) 1000 : -8 (min value) ... 1110 : -2 1111 : -1 So when you add 1 to 0111 (max) it goes to 1000 , and that's the minimum. Expand this

Unsigned long long overflow error?

纵然是瞬间 提交于 2019-12-10 18:25:56
问题 I have been having some strange issues with unsigned long long. It happens when I set an unsigned long long (I used size_t, however the problem is repeatable with u-l-l). I have set it to 2^31, however for some reason it reverts to 18446744071562067968, or 2^64 - 2^31. Keep in mind I am using an x64 compilation: unsigned long long a = 1 << 31; cout << a; //Outputs 18446744071562067968, Expected 2147483648 I thought the limits of u-l-l were 2^64-1? So why can 2^31 not be stored? 2^30 works

Why/how does gcc compile the undefined behaviour in this signed-overflow test so it works on x86 but not ARM64?

牧云@^-^@ 提交于 2019-12-10 18:01:13
问题 I was self-studying CSAPP and got a strange result when I ran into a strange issue during the run of a assertion test. I'm not sure what to start this question with, so let me get the code first (file name visible in comments): // File: 2.30.c // Author: iBug int tadd_ok(int x, int y) { if ((x ^ y) >> 31) return 1; // A positive number and a negative integer always add without problem if (x < 0) return (x + y) < y; if (x > 0) return (x + y) > y; // x == 0 return 1; } // File: 2.30-test.c //