I only have expreience with calling C or C++ functions from MATLAB. It looks to me like the only difference between calling functions in a shared library and calling functions from a MEX file is that with a shared library, you have to call the function with 'calllib' which is a command line type function and MEX functions allow you to call functions as if they are built-in functions so the interface is a little cleaner.
My suggestion is to use MEX files if
- You are using C++ (you may have to write a wrapper to use a C++ in a shared library)
- You are using MATLAB as the glue for a large number of optimized C or C++ routines. You'll want to be able to call them cleanly.
Use shared library if
- You already have an existing C library that can be used without modification.
- You only need a small number of calls to C functions.
Really, it comes down to the interface. I personally prefer the MEX file route because it provides the cleanest interface from MATLAB to your C or C++ function. You can call it like just another function with standard MATLAB types. With a shared library, you may have to do some data formatting before calling the library function
