recently I installed Microsoft R Open 3.3.1 on my MacBook Pro (El Capitan 10.11.6). When I open RStudio or I use the R console I get this warning message:
I had the issue also. The issue had been solved after installing XQuartz and then re-logining the system.
Note: the use of X11 (including tcltk) requires XQuartz to be installed since it is no longer part of OS X. Always re-install XQuartz when upgrading your OS X to a new major version.