I am used to building lists in android using adapters. If I need some long-to-get data, I use an asynctask, or a simple runnable, to update the data structure on which the adapt
One problem your code has which loaders aim to fix is what happens if your activity is restarted (say due to device rotation or config change) while your async task is still in progress? in your case your restarted activity will start a 2nd instance of the task and throw away the results from the first one. When the first one completes you can end up with crashes due to the fact your async task has a reference is what is now a finished activity.
And yes using loaders often makes for more/more complex code, particularly if you can't use one of the provided loaders.