Let's just concentrate on the answer to the question, which started with a "Why": Because you didn't turn optimisations on, and Swift relies heavily on compiler optimisation.
That said, doing image processing in C is truly daft. That's what you have CGImage and friends for.
