If it is not possible to do / complete a minor collection, then a major / full collection is performed. This is typically done using a mark-sweep-compact algorithm rather than copying algorithm ... which is one reason why full collection is expensive.
But ultimately (if you keep filling the heap) a full collection will not be able to reclaim enough space to continue and an OOME will be thrown. (Or if you are using -XX:+UseGCOverheadLimit, the OOME will be thrown when the percentage time spent in GC exceeds a designated threshold.)
