You are likely running a server version of MongoDB 3.0 or above ( 3.x series current as writing ), which has an updated security authentication mechanism ( SCRAM-SHA-1 from MONGODB-CR ) that is not compatible with older driver versions that do not support it.
The as of current rmongodb package release ( version 1.8.0 of writing ), this driver is based on the legacy C driver implementation which is not compatible with the new authentication methods. As is also noted in the issues on that repository, the author notes this driver dependency and states that the package would require a rewrite to utilize the new API that supports the new authentication method.
As of writing, there does not appear to be any moves to make any such changes, aside for establishing a new branch which is not presently ready for release.
Your options therefore presently are:
- Work without authentication where possible
- Downgrade the MongoDB server version to one that supports the old authentication
- Look for other driver implementations that support the new authentication.
So "rmongodb" itself cannot currently connect to MongoDB 3.x servers. Either apply one of the other choices, and/or contribute to the respository yourself if you are able to speed it's development into the next version with full authentication suppport.
Other possible driver alternatives are linked or dicussed in the issue linked in this answer.