Android Studio: Update/Remove Polylines or Points without calling Google Directions-API again

Question

I\'ve created a simple app using google maps and it\'s API. I\'ve already done plotting my route from Origin to Destination, including the markers.

Markers Plotting