Dragging annotations to the left edge is causing issues

Question

I have this JSFiddle, where I can draw annotations and drag them around.

But, in case I draw a segment or arrow-segment and in case I try to