I do understand that similar issues have been reported here but mine is somewhat different.
I have recently upgraded the com.android.support:appcompat-v7 to v21 and
Incase anyone lands here facing the same issue, the behavior occurs because "they" (Google) have changed the behavior of adjustPan moving forward from Android L.
To fix the issue, I am using a keyboard util in the library as indicated here.
A note from the developer is that this fix is to be used only in fragment/activity becaues this uses a GlobalLayoutListener which lowers the frameRate a bit but this is the only option to get it working