http://clb.demon.fi/files/RectangleBinPack.pdf is key. That is the reference on 2d bin packing.
You might be able to modify one of the algorithms in there to satisfy your need. I doubt the rotation is needed, the algorithms are pretty advanced as they are.
This (https://github.com/Lalaland/PixelPacker/blob/master/src/algoMaxRects.cpp) is an example of how to implement the MaxRects algorithm.
The modification you would probably have to make is at the top of the algorithm, when selecting the next rectangle to use. Simply have it also look at the different orientation of the rectangles along with cycling through the whole list.
