Even I was facing the same problem with making ViewPager vertical by swapping X and Y. It was not at all smooth.
That's because it used to work only when the angle of swiping was less than 7.125 degrees = arctan (1/8) while intercept and 14 degrees = arctan (1/4) while touch; as original ViewPager horizontal one work when the angle of swiping is less than 26.565 degrees = arctan (1/2) while intercept and 45 degrees = arctan (1) while touch.
That's why I copied the code of Android support-core-ui and moved the values to variables, which I multiplied using reflection.
Please find the code and README at https://github.com/deepakmishra/verticalviewpager
and VerticalViewPager at https://github.com/deepakmishra/verticalviewpager/blob/master/app/src/main/java/com/viewpager/VerticalViewPager.java