Sounds like you are looking for somekind of heuristics for vectorization of pixel data. There is no such thing built into Qt. I would suggest researching vectorization.
The best I can suggest is that if you are also writing the drawing tool, then trace the path as the user draws it and save it as QPicture, that way it is never on pixel form.