Pattern for generating negative Scalacheck scenarios: Using property based testing to test validation logic in Scala

Question

We are looking for a viable design pattern for building Scalacheck Gen (generators) that can produce both positive and negative test scenarios. This will allow us to run forAll tests to validate functionality (positive cases), and also verify that our case class validation works correctly by failing on all invalid combinations of data.

Making a simple, parameterized Gen that does this on a one-off basis is pretty easy. For example:

  def idGen(valid: Boolean = true): Gen[String] = Gen.oneOf(ID.values.toList).map(s => if (valid) s else Gen.oneOf(simpleRandomCode(4), "").sample.get)

With the above, I can get a valid or invalid ID for testing purposes. The valid one, I use to make sure business logic succeeds. The invalid one, I use to make sure our validation logic rejects the case class.

Ok, so -- problem is, on a large scale, this becomes very unwieldly. Let's say I have a data container with, oh, 100 different elements. Generating a "good" one is easy. But now, I want to generate a "bad" one, and furthermore:

1. I want to generate a bad one for each data element, where a single data element is bad (so at minimum, at least 100 bad instances, testing that each invalid parameter is caught by validation logic).

2. I want to be able to override specific elements, for instance feeding in a bad ID or a bad "foobar." Whatever that is.

One pattern we can look to for inspiration is apply and copy, which allows us to easily compose new objects while specifying overridden values. For example:

val f = Foo("a", "b") // f: Foo = Foo(a,b)
val t = Foo.unapply(f) // t: Option[(String, String)] = Some((a,b))
Foo(t.get._1, "c") // res0: Foo = Foo(a,c)

Above we see the basic idea of creating a mutating object from the template of another object. This is more easily expressed in Scala as:

val f = someFoo copy(b = "c")

Using this as inspiration we can think about our objectives. A few things to think about:

1. First, we could define a map or a container of key/values for the data element and generated value. This could be used in place of a tuple to support named value mutation.

2. Given a container of key/value pairs, we could easily select one (or more) pairs at random and change a value. This supports the objective of generating a data set where one value is altered to create failure.

3. Given such a container, we can easily create a new object from the invalid collection of values (using either apply() or some other technique).

4. Alternatively, perhaps we can develop a pattern that uses a tuple and then just apply() it, kind of like the copy method, as long as we can still randomly alter one or more values.

We can probably explore developing a reusable pattern that does something like this:

def thingGen(invalidValueCount: Int): Gen[Thing] = ???
def someTest = forAll(thingGen) { v => invalidV = v.invalidate(1); validate(invalidV) must beFalse }

In the above code, we have a generator thingGen that returns (valid) Things. Then for all instances returned, we invoke a generic method invalidate(count: Int) which will randomly invalidate count values, returning an invalid object. We can then use that to ascertain whether our validation logic works correctly.

This would require defining an invalidate() function that, given a parameter (either by name, or by position) can then replace the identified parameter with a value that is known to be bad. This implies have an "anti-generator" for specific values, for instance, if an ID must be 3 characters, then it knows to create a string that is anything but 3 characters long.

Of course to invalidate a known, single parameter (to inject bad data into a test condition) we can simply use the copy method:

def thingGen(invalidValueCount: Int): Gen[Thing] = ???
def someTest = forAll(thingGen) { v => v2 = v copy(id = "xxx"); validate(v2) must beFalse }

That is the sum of my thinking to date. Am I barking up the wrong tree? Are there good patterns out there that handle this kind of testing? Any commentary or suggestions on how best to approach this problem of testing our validation logic?

Answer 1

We can combine a valid instance and an set of invalid fields (so that every field, if copied, would cause validation failure) to get an invalid object using shapeless library.

Shapeless allows you to represent your class as a list of key-value pairs that are still strongly typed and support some high-level operations, and converting back from this representation to your original class.

In example below I'll be providing an invalid instance for each single field provided

import shapeless._, record._
import shapeless.labelled.FieldType
import shapeless.ops.record.Updater

A detailed intro

Let's pretend we have a data class, and a valid instance of it (we only need one, so it can be hardcoded)

case class User(id: String, name: String, about: String, age: Int) {
  def isValid = id.length == 3 && name.nonEmpty && age >= 0
}
val someValidUser = User("oo7", "Frank", "A good guy", 42)
assert(someValidUser.isValid)

We can then define a class to be used for invalid values:

case class BogusUserFields(name: String, id: String, age: Int)
val bogusData = BogusUserFields("", "1234", -5)

Instances of such classes can be provided using ScalaCheck. It's much easier to write a generator where all fields would cause failure. Order of fields doesn't matter, but their names and types do. Here we excluded about from User set of fields so we can do what you asked for (feeding only a subset of fields you want to test)

We then use LabelledGeneric[T] to convert User and BogusUserFields to their corresponding record value (and later we will convert User back)

val userLG = LabelledGeneric[User]
val bogusLG = LabelledGeneric[BogusUserFields]

val validUserRecord = userLG.to(someValidUser)
val bogusRecord = bogusLG.to(bogusData)

Records are lists of key-value pairs, so we can use head to get a single mapping, and the + operator supports adding / replacing field to another record. Let's pick every invalid field into our user one at a time. Also, here's the conversion back in action:

val invalidUser1 = userLG.from(validUserRecord + bogusRecord.head)// invalid name
val invalidUser2 = userLG.from(validUserRecord + bogusRecord.tail.head)// invalid ID
val invalidUser3 = userLG.from(validUserRecord + bogusRecord.tail.tail.head) // invalid age

assert(List(invalidUser1, invalidUser2, invalidUser3).forall(!_.isValid))

Since we basically are applying the same function (validUserRecord + _) to every key-value pair in our bogusRecord, we can also use map operator, except we use it with an unusual - polymorphic - function. We can also easily convert it to List, because every element will be of a same type now.

object polymerge extends Poly1 {
  implicit def caseField[K, V](implicit upd: Updater[userLG.Repr, FieldType[K, V]]) =
    at[FieldType[K, V]](upd(validUserRecord, _))
}

val allInvalidUsers = bogusRecord.map(polymerge).toList.map(userLG.from)
assert(allInvalidUsers == List(invalidUser1, invalidUser2, invalidUser3))

Generalizing and removing all the boilerplate

Now the whole point of this was that we can generalize it to work for any two arbitrary classes. The encoding of all relationships and operations is a bit cumbersome and it took me a while to get it right with all the implicit not found errors, so I'll skip the details.

class Picks[A, AR <: HList](defaults: A)(implicit lgA: LabelledGeneric.Aux[A, AR]) {
  private val defaultsRec = lgA.to(defaults)

  object mergeIntoTemplate extends Poly1 {
    implicit def caseField[K, V](implicit upd: Updater[AR, FieldType[K, V]]) =
      at[FieldType[K, V]](upd(defaultsRec, _))
  }

  def from[B, BR <: HList, MR <: HList, F <: Poly](options: B)
    (implicit
      optionsLG: LabelledGeneric.Aux[B, BR],
      mapper: ops.hlist.Mapper.Aux[mergeIntoTemplate.type, BR, MR],
      toList: ops.hlist.ToTraversable.Aux[MR, List, AR]
    ) = {
    optionsLG.to(options).map(mergeIntoTemplate).toList.map(lgA.from)
  }
}

So, here it is in action:

val cp = new Picks(someValidUser)
assert(cp.from(bogusData) == allInvalidUsers)

Unfortunately, you cannot write new Picks(someValidUser).from(bogusData) because implicit for mapper requires a stable identifier. On the other hand, cp instance can be reused with other types:

case class BogusName(name: String)
assert(cp.from(BogusName("")).head == someValidUser.copy(name = ""))

And now it works for all types! And bogus data is required to be any subset of class fields, so it will work even for class itself

case class Address(country: String, city: String, line_1: String, line_2: String) {
  def isValid = Seq(country, city, line_1, line_2).forall(_.nonEmpty)
}

val acp = new Picks(Address("Test country", "Test city", "Test line 1", "Test line 2"))
val invalidAddresses = acp.from(Address("", "", "", ""))
assert(invalidAddresses.forall(!_.isValid))

You can see the code running at ScalaFiddle