How to use Jupyter + SparkR and custom R install

Question

I am using a Dockerized image and Jupyter notebook along with SparkR kernel. When I create a SparkR notebook, it uses an install of Microsoft R (3.3.2) instead of vanilla C

Answer 1

To use a custom R environment I believe you need to set the following application properties when you start Spark:

    "spark.r.command": "/custom/path/bin/R",
    "spark.r.driver.command": "/custom/path/bin/Rscript",
    "spark.r.shell.command" : "/custom/path/bin/R"

This is more completely documented here: https://spark.apache.org/docs/latest/configuration.html#sparkr