Flooding is a different problem from spam. You should definitely build the logic around rate limiting into your application, you can do this using validation to check that the user hasn't, for example, placed more than 2 orders in the last 15 minutes.
In regards to captchas any of the plugins you select are most likely going to be great. I wouldn't think of having to install RMagick as a positive or negative, it really isn't that hard to get working. If it was me choosing, my first instinct would be to go with recaptcha, it's the least annoying of them all.
Spam is another issue, it's often entered by human users who can bypass your captcha. Akismet is great for catching spam, definitely take a look at it, you can use it in conjunction with something like recaptcha.
Finally, modern bots are very sophisticated. Far more sophisticated than any of us probably expect. They can fully automate browsers, use OCR to read captcha text and generate spammy content that will bypass even the most sophisticated filters. That said, it's not about "stopping all spam/bots" it's about making the barrier to entry just high enough that it isn't worth it for the casual user.
