Is there some way to obfuscate C-based executables or libraries to prevent decompilation?
I think if you speak about compiled binary there is not much you can do, (perhaps only apply UPX or related tool) which does not make a lot of sence since it can be reversed.
If you talk about writing new code, try Self Modyfing C Code which will probably be the hardest way to re engineer your application.
Decompilation (No More Gotos) and both obfuscation practice (Flowtables) and theory (Indistinguishability Obfuscation) are active areas of research and therefore there are no solutions - only tools, techniques and expertise. If you really want your code to be impervious to decomplilation, create a web app, and put the sensitive code server side. But if you're stuck to the model of giving someone a binary, then you must wisely judge the trade-off you want to make between security and performance. Obfuscation comes at a cost, and still is never perfect. Some options
Don't be discouraged by Barak et al's seminal work on the impossibility of black box obfuscation. He only proves the impossibility of black box obfuscators, not the impossibility of many practical and worthwhile obfuscations. (Black box obfuscation being the inner workings of the program are completely unintelligible) Also don't be discouraged by pirates. There's always people who make it a point to buy your product if it is good.
No. You can make it more difficult to decompile but you cannot prevent it. My advice is to stop wasting your time and instead concentrate on delivering a fantastic product with ever-improving features.
Then people will be willing to pay for it.
Your main problem is that the only way to make your code un-decipherable is to make it un-runnable. Anything that can be loaded into a PC can be cracked. The people that do reverse engineering for fun, profit or fame are generally very good at it and will really not be the least bit phased by anything you do to try and stop them.
They have access to tools that make the job of deciphering your code far easier than the job you will have obfuscating it :-) Far better to convince the world at large that your software is worth buying, and seeing piracy as an opportunity to possibly convert "thieves" to genuine users.
For example, find out why they're not paying for your software and try to fix that. You'll never convert 100% of the people, some will pirate your code just for the fun of it.
Check out the series of articles running over on techdirt concerning CwF+RtB (connect with fans plus reason to buy). I've found many of the points raised there could be applicable to the software industry.
"obfuscated executables" makes no sense. The hardware has to be able to "understand" the code in able to to execute it, and it the hardware can understand it, a reverse engineering human can understand it. The most you can do will be make it more tedious to understand, but probably not by much, and at a cost.
compile with full optimization.
The easy way: Buy a packer/cryptor/obfuscator product. Some are expensive and used in games, some are not. Google for them by buzzwords like "copy protection", etc.
The fast way: pack with UPX and then mangle the header somewhere so it will still be loaded in memory and run fine, but the upx utility will fail with an error (try the version field). 95% will give up if the upx utility fails.
The hard way: Write your own packer.
oh, I forgot:
The real easy way: Just ship it as it is. No really - whatever you do people can still reverse engineer your code. The amount of effort you put it in just restricts how many can reverse it.