You can try -nodefaultlibs to avoid all the default libraries, then use -L and -l to add-back the libraries you want in the directories you want.
Directories specified on the command-line with the -L option should have priority over the default directories.