I have an iPython notebook that is completely made up of R cells. At the start of every cell, I have defined my R cell magic as follows:
%%R
IRKernel is an R kernel for ipython/jupyter as suggested by @cel. There are lots of other kernels that you can use.