I am using Jquery UI Drag and Drop ( http://jqueryui.com/demos/draggable ) together with https://github.com/furf/jquery-ui-touch-punch to map touch events over to mouse events.
It depends on your design of course, but you could try using handles.
Here is jQuery UI's documentation example :
drag with handle drag from content not drag with handle $("#draggable").draggable({handle:"p"}); $("#draggable2").draggable({cancel:"p.ui-widget-header"});
drag with handle
drag from content
not drag with handle
Or at least the option cancel could give you a start.
http://jsfiddle.net/Dn9DX/