Make ScalaCheck tests deterministic

Question

I would like to make my ScalaCheck property tests in my specs2 test suite deterministic, temporarily, to ease debugging. Right now, different values could be generated each time I re-run the test suite, which makes debugging frustrating, because you don't know if a change in observed behaviour is caused by your code changes, or just by different data being generated.

How can I do this? Is there an official way to set the random seed used by ScalaCheck?

I'm using sbt to run the test suite.

Bonus question: Is there an official way to print out the random seed used by ScalaCheck, so that you can reproduce even a non-deterministic test run?

Answer 1

If you're using pure ScalaCheck properties, you should be able to use the Test.Params class to change the java.util.Random instance which is used and provide your own which always return the same set of values:

def check(params: Test.Parameters, p: Prop): Test.Result

[updated]

I just published a new specs2-1.12.2-SNAPSHOT where you can use the following syntax to specify your random generator:

case class MyRandomGenerator() extends java.util.Random {
  // implement a deterministic generator 
}

"this is a specific property" ! prop { (a: Int, b: Int) =>
  (a + b) must_== (b + a)
}.set(MyRandomGenerator(), minTestsOk -> 200, workers -> 3)

Answer 2

As a general rule, when testing on non-deterministic inputs you should try to echo or save those inputs somewhere when there's a failure.

If the data is small, you can include it in the label or error message that gets shown to the user; for example, in an xUnit-style test: (since I'm new to Scala syntax)

testLength(String x) {
    assert(x.length > 10, "Length OK for '" + x + "'");
}

If the data is large, for example an auto-generated DB, you might either store it in a non-volatile location (eg. /tmp with a timestamped name) or show the seed used to generate it.

The next step is important: take that value, or seed, or whatever, and add it to your deterministic regression tests, so that it gets checked every time from now on.

You say you want to make ScalaCheck deterministic "temporarily" to reproduce this issue; I say you've found a buggy edge-case which is well-suited to becoming a unit test (perhaps after some manual simplification).

Answer 3

Bonus question: Is there an official way to print out the random seed used by ScalaCheck, so that you can reproduce even a non-deterministic test run?

From specs2-scalacheck version 4.6.0 this is now a default behaviour:

Given the test file HelloSpec:

package example

import org.specs2.mutable.Specification
import org.specs2.ScalaCheck

class HelloSpec extends Specification  with ScalaCheck {
package example

import org.specs2.mutable.Specification
import org.specs2.ScalaCheck

class HelloSpec extends Specification  with ScalaCheck {
  s2"""
    a simple property       $ex1
  """

  def ex1 = prop((s: String) => s.reverse.reverse must_== "")
}

build.sbt config:

import Dependencies._

ThisBuild / scalaVersion     := "2.13.0"
ThisBuild / version          := "0.1.0-SNAPSHOT"
ThisBuild / organization     := "com.example"
ThisBuild / organizationName := "example"

lazy val root = (project in file("."))
  .settings(
    name := "specs2-scalacheck",
    libraryDependencies ++= Seq(
      specs2Core,
      specs2MatcherExtra,
      specs2Scalacheck
    ).map(_ % "test")
  )

project/Dependencies:

import sbt._

object Dependencies {
  lazy val specs2Core                       = "org.specs2"             %% "specs2-core"               % "4.6.0"
  lazy val specs2MatcherExtra               = "org.specs2"             %% "specs2-matcher-extra"      % specs2Core.revision
  lazy val specs2Scalacheck                 = "org.specs2"             %% "specs2-scalacheck"         % specs2Core.revision

}

When you run the test from the sbt console:

sbt:specs2-scalacheck> testOnly example.HelloSpec

You get the following output:

[info] HelloSpec
[error]     x a simple property
[error]  Falsified after 2 passed tests.
[error]  > ARG_0: "\u0000"
[error]  > ARG_0_ORIGINAL: "猹"
[error]  The seed is X5CS2sVlnffezQs-bN84NFokhAfmWS4kAg8_gJ6VFIP=
[error]  
[error]  > '' != '' (HelloSpec.scala:11)
[info] Total for specification HelloSpec

To reproduce that specific run (i.e with the same seed)You can take the seed from the output and pass it using the command line scalacheck.seed:

sbt:specs2-scalacheck>testOnly example.HelloSpec -- scalacheck.seed X5CS2sVlnffezQs-bN84NFokhAfmWS4kAg8_gJ6VFIP=

And this produces the same output as before.

You can also set the seed programmatically using setSeed:

def ex1 = prop((s: String) => s.reverse.reverse must_== "").setSeed("X5CS2sVlnffezQs-bN84NFokhAfmWS4kAg8_gJ6VFIP=")

Yet another way to provide the Seed is pass an implicit Parameters where the seed is set:

package example

import org.specs2.mutable.Specification
import org.specs2.ScalaCheck
import org.scalacheck.rng.Seed
import org.specs2.scalacheck.Parameters

class HelloSpec extends Specification  with ScalaCheck {

  s2"""
    a simple property       $ex1
  """

  implicit val params = Parameters(minTestsOk = 1000, seed = Seed.fromBase64("X5CS2sVlnffezQs-bN84NFokhAfmWS4kAg8_gJ6VFIP=").toOption)

  def ex1 = prop((s: String) => s.reverse.reverse must_== "")
}

Here is the documentation about all those various ways. This blog also talks about this.

Answer 4

For scalacheck-1.12 this configuration worked:

new Test.Parameters {
  override val rng = new scala.util.Random(seed)
}

For scalacheck-1.13 it doesn't work anymore since the rng method is removed. Any thoughts?