A transformation is nothing more than a matrix multiplying a vector to produce the transformed vector, so if you don't understand matrix multiplication and addition you can't get very far.
Start with matricies and linear algebra. There are lots of books out there, but realize that based on the statement that I made above you don't need to read that whole book. You won't need eigenvalues or Gaussian elimination or vector spaces or any of the other stuff that will be advanced and difficult.
You just need to know how to extend what you know about multiplying and adding numbers to matricies.
Getting the entries in that transformation matrix is another matter altogether. You'll need a good book on mathematics and computer graphics. You won't find that in a linear algebra textbook.
