Given a rectangular shape S, with aspect ratio sx/sy, and two other rectangular shapes A (with aspect ratio ax/ay) and B (with aspect ratio bx/by) how can I find out which of sh
Is it just whichever of (sx/sy)/(ax/ay) and (sx/sy)/(bx/by) is closest to 1?
That sounds reasonable. You could also just minimize the difference:
let target_ratio = sx/sy
let a_ratio = ax/ay
let b_ration = bx/by
if |target_ratio - a_ratio| < |target_ratio - b_ratio|
a_ratio is closer to target
else
b_ratio is closer to target
Update: the algorithm in this answer does not quite work, as explained in the comments below. The OP updated his question to include the algorithm that he used, which works seems to work fine.