If you're using Matlab, is it taboo to link to Matlab from your C code? This is a starting point. You can actually ask the Matlab engine to do anything you can do from within the Matlab interface by calling it from C.
A Butterworth filter is recursive, so it should be implemented as an IIR filter. Once you have stable filter coefficients from your Matlab testing you can simply feed them to a generic IIR algorithm (pretty simple). You can approximate the filter response with a large FIR filter and convolution, which can be moved into frequency domain to solve phase problems, but it's not truly Butterworth if you do that.
I'd recommend building an algorithm from scratch as a learning exercise, but if you need it ASAP there's probably any number of libraries out there to help you. This came up in search.
