This is a typical problem in rJava. It is answered in the XLConnect documentation which also uses rJava to connect to excel the same way as the xlsx library. I quote from here:
"This is caused by the fact that XLConnect (same for xlsx) needs to copy your entire data object over to the JVM in order
to write it to a file and the JVM has to be initialized with a fixed upper limit on its memory size. To change
this amount, you can pass parameters to the R’s JVM just like you can to a command line Java process via
rJava’s options support:
options(java.parameters = "- Xmx1024m")
Note, however, that these parameters are evaluated exactly once per R session when the JVM is initialized
- this is usually once you load the first package that uses Java support, so you should do this as early as
possible."
As it is mentioned above run the options function at the beginning of your script before loading any libraries and if you are running it through Rstudio make sure you restart it before you run the script.
Also, please note that it is still not certain that even this will work depending on the size of the file you are trying to parse.
