I am using Jquery UI Drag and Drop ( http://jqueryui.com/demos/draggable ) together with https://github.com/furf/jquery-ui-touch-punch to map touch events over to mouse events.
It depends on your design of course, but you could try using handles.
Here is jQuery UI's documentation example :
<div id="draggable" class="ui-widget-content">
<p class="ui-widget-header">drag with handle</p>
</div>
<div id="draggable2" class="ui-widget-content">
<p>drag from content</p>
<p class="ui-widget-header">not drag with handle</p>
</div>
$("#draggable").draggable({handle:"p"});
$("#draggable2").draggable({cancel:"p.ui-widget-header"});
Or at least the option cancel could give you a start.
http://jsfiddle.net/Dn9DX/
I solved this by switching from https://github.com/furf/jquery-ui-touch-punch to the code in this solution: Javascript Drag and drop for touch devices.
In the end all I needed to adjust was to remove the event.preventDefault();
to re-enable scolling.
Edit
Basically I used the code from the second answer I linked - with some adjustments. Here is the JS fiddle of my solution, hope it helps: http://jsfiddle.net/LQuyr/8/