I am designing a system in C/C++ which is extendible with all sort of plugins. There is a well defined C public API which mostly works with (const) char*
and other
Simply put, you can't do that in the same process. If your plugins are written in C or C++, they can contain numerous sources of undefined behavior, meaning sources for undetectable unavoidable crashes. So you should either launch the plugins in their own processes like kassak suggested and let them crash if they want to, or use another language for your plugins, e.g. some intepreted scripting language like lua.
Have a look at http://msdn.microsoft.com/en-us/library/1deeycx5(v=vs.90).aspx
I use /EHa in one of my projects to help me catch exceptions from libraries that do stupid things. If you compile your code with this setting a normal try catch block will catch exceptions like devide by zero, etc.
Not sure if there is some equivalent for this on Linux -- please let me know if there is..
Make a wrapper process for plugin and communicate with that wrapper through IPC. In case of plugin failure your main process would be untouched