On the first iteration of the loop, i
is 0, so bignum % i
is a division by 0, which is what causes the floating point exception. The same happens with j
in your inner loop.
Since you're testing for primality, dividing by 0 or by 1 doesn't make sense, so start with 2. Also, you should stop at sqrt(bignum)
.
uint64_t limit = sqrt(bignum);
for (i = 2; i < limit; i++) {
...
for (j = 2; j < i; j++) {