I have implemented an openCV applicationWhere I use SURF descriptor. It is working fine the code looks like this:
I reduce the input video stream size to
Do you need to use the SURF feature/descriptor for your application? SURF is attractive as it matches very nicely, but as you've found out it is somewhat slow. If you're just tracking points through a video you could make the assumption that points will not vary much frame-to-frame and so you could detect and match Harris/FAST corners and then filter matches to be valid only if they're within an x-pixel radius of the original point?
OpenCV has an (albeit somewhat limited) selection of feature detectors and descriptor extractors and descriptor matchers, it would be worth investigating the options if you've not already.