Failing a scalatest when akka actor throws exception outside of the test thread

Question

I\'ve had a situation come up and bite me a few times where I\'m testing an Actor and the Actor throws an exception unexpectedly (due to a bug), but the test still passes.

Answer 1

Okay, I've had a little time to play with this. I've got a nice solution that uses an event listener and filter to catch errors. (Checking isTerminated or using TestProbes is probably good in more focused cases but seems awkward when trying to make something to mix into any old test.)

import akka.actor.{Props, Actor, ActorSystem}
import akka.event.Logging.Error
import akka.testkit._
import com.typesafe.config.Config
import org.scalatest._
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.mock.EasyMockSugar
import scala.collection.mutable

trait AkkaErrorChecking extends ShouldMatchers {
  val system:ActorSystem
  val errors:mutable.MutableList[Error] = new mutable.MutableList[Error]
  val errorCaptureFilter = EventFilter.custom {
    case e: Error =>
      errors += e
      false // don't actually filter out this event - it's nice to see the full output in console.
  }

  lazy val testListener = system.actorOf(Props(new akka.testkit.TestEventListener {
    addFilter(errorCaptureFilter)
  }))

  def withErrorChecking[T](block: => T) = {
    try {
      system.eventStream.subscribe(testListener, classOf[Error])
      filterEvents(errorCaptureFilter)(block)(system)
      withClue(errors.mkString("Akka error(s):\n", "\n", ""))(errors should be('empty))
    } finally {
      system.eventStream.unsubscribe(testListener)
      errors.clear()
    }
  }
}

You can just use withErrorChecking inline at specific spots, or mix it into a Suite and use withFixture to do it globally across all tests, like this:

trait AkkaErrorCheckingSuite extends AkkaErrorChecking with FunSpec {
  override protected def withFixture(test: NoArgTest) {
    withErrorChecking(test())
  }
}

If you use this in my original example, then you will get the first test "calls foo only when it receives a message" to fail, which is nice because that's where the real failure is. But the downstream test will still fail as well due to the system blowing up. To fix that, I went a step further and used a fixture.Suite to instance a separate TestKit for each test. That solves lots of other potential test isolation issues when you have noisy actors. It requires a little more ceremony declaring each test but I think it's well worth it. Using this trait with my original example I get the first test failing and the second one passing which is just what I want!

trait IsolatedTestKit extends ShouldMatchers { this: fixture.Suite =>
  type FixtureParam = TestKit
  // override this if you want to pass a Config to the actor system instead of using default reference configuration
  val actorSystemConfig: Option[Config] = None

  private val systemNameRegex = "[^a-zA-Z0-9]".r

  override protected def withFixture(test: OneArgTest) {
    val fixtureSystem = actorSystemConfig.map(config => ActorSystem(systemNameRegex.replaceAllIn(test.name, "-"), config))
                                         .getOrElse    (ActorSystem (systemNameRegex.replaceAllIn(test.name, "-")))
    try {
      val errorCheck = new AkkaErrorChecking {
        val system = fixtureSystem
      }
      errorCheck.withErrorChecking {
        test(new TestKit(fixtureSystem))
      }
    }
    finally {
      fixtureSystem.shutdown()
    }
  }
}