Failing a scalatest when akka actor throws exception outside of the test thread

Question

I\'ve had a situation come up and bite me a few times where I\'m testing an Actor and the Actor throws an exception unexpectedly (due to a bug), but the test still passes.

Answer 1

Other than examining the logs, I can think of two ways to fail tests when an actor crashes:

• Ensure that no Terminated message is received
• Check the TestActorRef.isTerminated property

The latter option is deprecated, so I'll ignore it.

Watching Other Actors from Probes describes how to setup a TestProbe. In this case it might look something like:

val probe = TestProbe()
probe watch ref

// Actual test goes here ...

probe.expectNoMessage()

If the actor dies due to an exception it will generate the Terminated message. If that happens during the test and you expect something else, the test will fail. If it happens after your last message expectation, then the expectNoMessage() should fail when Terminated is received.

Answer 2

Okay, I've had a little time to play with this. I've got a nice solution that uses an event listener and filter to catch errors. (Checking isTerminated or using TestProbes is probably good in more focused cases but seems awkward when trying to make something to mix into any old test.)

import akka.actor.{Props, Actor, ActorSystem}
import akka.event.Logging.Error
import akka.testkit._
import com.typesafe.config.Config
import org.scalatest._
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.mock.EasyMockSugar
import scala.collection.mutable

trait AkkaErrorChecking extends ShouldMatchers {
  val system:ActorSystem
  val errors:mutable.MutableList[Error] = new mutable.MutableList[Error]
  val errorCaptureFilter = EventFilter.custom {
    case e: Error =>
      errors += e
      false // don't actually filter out this event - it's nice to see the full output in console.
  }

  lazy val testListener = system.actorOf(Props(new akka.testkit.TestEventListener {
    addFilter(errorCaptureFilter)
  }))

  def withErrorChecking[T](block: => T) = {
    try {
      system.eventStream.subscribe(testListener, classOf[Error])
      filterEvents(errorCaptureFilter)(block)(system)
      withClue(errors.mkString("Akka error(s):\n", "\n", ""))(errors should be('empty))
    } finally {
      system.eventStream.unsubscribe(testListener)
      errors.clear()
    }
  }
}

You can just use withErrorChecking inline at specific spots, or mix it into a Suite and use withFixture to do it globally across all tests, like this:

trait AkkaErrorCheckingSuite extends AkkaErrorChecking with FunSpec {
  override protected def withFixture(test: NoArgTest) {
    withErrorChecking(test())
  }
}

If you use this in my original example, then you will get the first test "calls foo only when it receives a message" to fail, which is nice because that's where the real failure is. But the downstream test will still fail as well due to the system blowing up. To fix that, I went a step further and used a fixture.Suite to instance a separate TestKit for each test. That solves lots of other potential test isolation issues when you have noisy actors. It requires a little more ceremony declaring each test but I think it's well worth it. Using this trait with my original example I get the first test failing and the second one passing which is just what I want!

trait IsolatedTestKit extends ShouldMatchers { this: fixture.Suite =>
  type FixtureParam = TestKit
  // override this if you want to pass a Config to the actor system instead of using default reference configuration
  val actorSystemConfig: Option[Config] = None

  private val systemNameRegex = "[^a-zA-Z0-9]".r

  override protected def withFixture(test: OneArgTest) {
    val fixtureSystem = actorSystemConfig.map(config => ActorSystem(systemNameRegex.replaceAllIn(test.name, "-"), config))
                                         .getOrElse    (ActorSystem (systemNameRegex.replaceAllIn(test.name, "-")))
    try {
      val errorCheck = new AkkaErrorChecking {
        val system = fixtureSystem
      }
      errorCheck.withErrorChecking {
        test(new TestKit(fixtureSystem))
      }
    }
    finally {
      fixtureSystem.shutdown()
    }
  }
}

Answer 3

Thinking in Actors there is also another solution: failures travel to the supervisor, so that is the perfect place to catch them and feed them into the test procedure:

val failures = TestProbe()
val props = ... // description for the actor under test
val failureParent = system.actorOf(Props(new Actor {
  val child = context.actorOf(props, "child")
  override val supervisorStrategy = OneForOneStrategy() {
    case f => failures.ref ! f; Stop // or whichever directive is appropriate
  }
  def receive = {
    case msg => child forward msg
  }
}))

You can send to the actor under test by sending to failureParent and all failures—expected or not—go to the failures probe for inspection.