I'm not too knowledgeable about exactly how it's done, but I know that modifying a VBO from a shader is called Transform Feedback. You'll need OpenGL >=3.0, or OpenGL >= 2.0 with the NV_transform_feedback
or EXT_transform_feedback
extension.
If you need more general computation power, you could try doing some OpenCL/OpenGL interop...