Hi please forgive me if I am mistaking when explaining the problem . I have a Custom Base adapter in which there is two imageView and two TextView and I am usning an Async task
I would suggest you to try Picasso to remove the hassle of downloading images/caching etc. Once you have the jar in your workspace, all you need to do in the getView() method of the adapter class is this.
Picasso.with(context).load(result.get(IMAGE_URL)).into(holder.shared_image);
No need of the DownloadAsyncTask. As @zapl also suggested in the comment, there are other libraries like Volley and UniversalImageLoader, but I liked Picasso. You can also easily apply transformations like rounded-image using the Transformation interface provided by Picasso.