testthat fails within devtools::check but works in devtools::test

Question

Is there any way to reproduce the environment which is used by devtools::check?

I have the problem that my tests work with devtools::test() but fail within devtools::check(). My problem is now, how to find the problem. The report of check just prints the last few lines of the error log and I can't find the complete report for the testing.

checking tests ... ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
...

I know that check uses a different environment compared to test but I don't know how I should debug these problems since they are not reproducible at all. Specially these test where running a few month ago, so not sure where to look for the problem.

EDIT

actually I tried to locate my problem and I found a solution. But to post my solution to it, I have to add more details.

So my test always failed since I was testing a markdown script if it is running without errors and afterwards I was checking if some of the environmental variables are set correctly. These where results which I calculate with the script as well as standard settings which I set. So I wanted to get a warning if I forgot to change some of my settings after developing...

Anyway, since it is a markdown script, I had to extract the code and I was using comments from this post knitr: run all chunks in an Rmarkdown document using knitr::purl to get the code and sys.source to execute it.

runAllChunks <- function(rmd, envir=globalenv()){
  # as found here https://stackoverflow.com/questions/24753969
  tempR <- tempfile(tmpdir = '.', fileext = ".R")
  on.exit(unlink(tempR))
  knitr::purl(rmd, output=tempR, quiet=TRUE)
  sys.source(tempR, envir=envir)
}

For some reason, this produces an error since maybe a few weeks (not sure which new packages I installed lately...). But since there is a new comment, that I can just use knitr::knit which also executes the code, this worked as expected and now my test no longer complains.

So in the end, I don't know where the problem exactly was, but this is now working.

Answer 1

I recently had a similar issue with my tests breaking (succeeding with devtools::test() but failing with devtools::check()). I don't know if this solution necessarily fixes the problem above, but it should help to track down similar problems.

In my case, the problem ultimately came down to using a function that needed a package listed in Suggests rather than in Imports/Depends. In particular, my function called httr::content(), which broke when I tried to pass it the as = "parsed" argument. It turns out that as = "parsed" uses a suggested package, readr to read a csv, and I needed to add it to my dependencies for devtools::check() to work.

Answer 2

This is a known issue with testthat. The workaround is to add the following as the 1st line in tests/testthat.R:

Sys.setenv(R_TESTS="")

Answer 3

So as I shortly mentioned above, I changed some of my code to no longer user knitr::purl but using knitr::knit and this solved my problem.

expect_error(f <- runAllChunks('010_main_lfq_analysis.Rmd'), NA)
expect_error(f <- knitr::knit('010_main_lfq_analysis.Rmd', output='jnk.R', quiet=TRUE, envir=globalenv()), NA)

Answer 4

In case it helps someone else, this is what worked for me

1. Re-install all relevant packages. E.g. install.packages("testthat", "dplyr", "lubridate", "stringr") (I included all packages my package uses)
2. Close RStudio and reopen

Then all tests passed