I have simple ListView with ArrayAdapter which is working just fine. The problems start with RTL language (Arabic in this case).
W
Try manually setting the direction based on configuration (that is quite reliably updated based on device language) by adding rowView.setLayoutDirection(getContext().getResources().getConfiguration().getLayoutDirection());
in the getView of your ListAdapter.
From: ListView's first entry always incorrect for RTL
This solved my problem that was similar to your.