You could start with a minimal linux distribution such as Ubuntu Server and install only X-Windows (without any Window Manager) on top of it: https://help.ubuntu.com/community/ServerGUI
Then, start your Qt application by adding a call to it in the .xinitrc initialization script.