Testing an assertion that something must not compile

Question

The problem

When I\'m working with libraries that support type-level programming, I often find myself writing comments like the following (from an example presente

Answer 1

Not a framework, but Jorge Ortiz (@JorgeO) mentioned some utilities he added to the tests for Foursquare's Rogue library at NEScala in 2012 which support tests for non-compilation: you can find examples here. I've been meaning to add something like this to shapeless for quite a while.

More recently, Roland Kuhn (@rolandkuhn) has added a similar mechanism, this time using Scala 2.10's runtime compilation, to the tests for Akka typed channels.

These are both dynamic tests of course: they fail at (test) runtime if something that shouldn't compile does. Untyped macros might provide a static option: ie. a macro could accept an untyped tree, type check it and throw a type error if it succeeds). This might be something to experiment with on the macro-paradise branch of shapeless. But not a solution for 2.10.0 or earlier, obviously.

Update

Since answering the question, another approach, due to Stefan Zeiger (@StefanZeiger), has surfaced. This one is interesting because, like the untyped macro one alluded to above, it is a compile time rather than (test) runtime check, however it is also compatible with Scala 2.10.x. As such I think it is preferable to Roland's approach.

I've now added implementations to shapeless for 2.9.x using Jorge's approach, for 2.10.x using Stefan's approach and for macro paradise using the untyped macro approach. Examples of the corresponding tests can be found here for 2.9.x, here for 2.10.x and here for macro paradise.

The untyped macro tests are the cleanest, but Stefan's 2.10.x compatible approach is a close second.

Answer 2

Do you know about partest in the Scala project? E.g. CompilerTest has the following doc:

/** For testing compiler internals directly.
* Each source code string in "sources" will be compiled, and
* the check function will be called with the source code and the
* resulting CompilationUnit. The check implementation should
* test for what it wants to test and fail (via assert or other
* exception) if it is not happy.
*/

It is able to check for example whether this source https://github.com/scala/scala/blob/master/test/files/neg/divergent-implicit.scala will have this result https://github.com/scala/scala/blob/master/test/files/neg/divergent-implicit.check

It's not a perfect fit for your question (since you don't specify your test cases in terms of asserts), but may be an approach and/or give you a head start.

Answer 3

ScalaTest 2.1.0 has the following syntax for Assertions:

assertTypeError("val s: String = 1")

And for Matchers:

"val s: String = 1" shouldNot compile

Answer 4

The compileError macro in µTest does just that:

compileError("true * false")
// CompileError.Type("value * is not a member of Boolean")

compileError("(}")
// CompileError.Parse("')' expected but '}' found.")

Answer 5

Based on the links provided by Miles Sabin I was able to use the akka version

import scala.tools.reflect.ToolBox

object TestUtils {

  def eval(code: String, compileOptions: String = "-cp target/classes"): Any = {
    val tb = mkToolbox(compileOptions)
    tb.eval(tb.parse(code))
  }

  def mkToolbox(compileOptions: String = ""): ToolBox[_ <: scala.reflect.api.Universe] = {
    val m = scala.reflect.runtime.currentMirror
    m.mkToolBox(options = compileOptions)
  }
}

Then in my tests I used it like this

def result = TestUtils.eval(
  """|import ee.ui.events.Event
     |import ee.ui.events.ReadOnlyEvent
     |     
     |val myObj = new {
     |  private val writableEvent = Event[Int]
     |  val event:ReadOnlyEvent[Int] = writableEvent
     |}
     |
     |// will not compile:
     |myObj.event.fire
     |""".stripMargin)

result must throwA[ToolBoxError].like {
  case e => 
    e.getMessage must contain("value fire is not a member of ee.ui.events.ReadOnlyEvent[Int]") 
}