How to add tracing within a 'for' comprehension?

Question

For log tracing inside a for comprehension, I\'ve used dummy assignment like this:

val ll = List(List(1,2),List(1))            

for {
  outer <         


        

Answer 1

The short answer to your question is the WriterT monad transformer. The long answer follows.

In the following explanation, I am going to give you a tool that achieves your desired goal, but using a very different mechanism to those that have already been stated. I will offer my brief opinion on the merits of the differences toward the end.

First, what is a for-comprehension? A for-comprehension is (approximately enough for our purposes) a monad comprehension but with a different name. This happens to be a common theme; C# has LINQ for example.

What is a monad?

For our purposes of explanation (this is not entirely true, but true enough for now), a monad is any value for M that implements the following trait:

trait Monad[M[_]] {
  def flatMap[A, B](a: M[A], f: A => M[B]): M[B]
  def map[A, B](a: M[A], f: A => B): M[B]
}

That is to say, if you have a Monad implementation for some M, then you are able to use a for-comprehension on values with the type M[A] for any value of A.

Some examples for values of M that would fit this interface and are in the standard library are List, Option and Parser. Of course, you probably use for-comprehensions from them all the time. Other examples might be your own data type. For example:

case class Inter[A](i: Int => A) 

...and here is the Monad implementation for Inter:

val InterMonad: Monad[Inter] = new Monad[Inter] {
  def flatMap[A, B](a: Inter[A], f: A => Inter[B]) =
    Inter(n => f(a.i(n)).i(n))
  def map[A, B](a: Inter[A], f: A => B) =
    Inter(n => f(a.i(n)))
}

There are many many more values for M. The question you have is, essentially, how do we add logging support to these values?

The Writer data type

The Writer data type is simply a pair (scala.Tuple2). In this pair, we compute some value (let's call it A) and associate another value with it (let's call it LOG).

// simply, a pair
case class Writer[LOG, A](log: LOG, value: A)

As we compute values we wish to append a log value to the currently computed log. Before we start computing anything, we wish to have an empty log. We can represent these operations (append and empty) in an interface:

trait Monoid[A] {
  def append(a1: A, a2: A): A
  def empty: A
}

There are some laws that all implementations of this interface must follow:

• Associativity: append(x, append(y, z)) == append(append(x, y), z)
• Right Identity: append(empty, x) == x
• Left Identity: append(x, empty) == x

As a side note, these are also the same laws that implementations of the Monad interface must follow, but I have left those out to save confusion and to stay on the point of logging.

There are many examples of implementations of this Monoid interface, one of which is List:

def ListMonoid[A]: Monoid[List[A]] = new Monoid[List[A]] {
  def append(a1: List[A], a2: List[A]) = 
    a1 ::: a2
  def empty =
    Nil
}

Simply to mark the point of how diverse this Monoid interface is, here is another example of an implementation:

def EndoMonoid[A]: Monoid[A => A] = new Monoid[A => A] {
  def append(a1: A => A, a2: A => A) =
    a1 compose a2
  def empty =
    a => a
}

I understand that these generalisations may be getting a bit difficult to keep in your head, so what I am going to do now, is specialise the Writer to use a List of String values for its log. Sounds reasonable enough? However, there are a couple points of note:

1. In practice we wouldn't use List because of the undesirable algorithmic complexity of its append. Rather we might use a finger-tree based sequence or something else with a faster insertion at the end operation.
2. List[String] is just one example of a Monoid implementation. It s important to keep in mind that there are an enormous number of other possible implementations, many of which are not collection types. Just remember that all we need is any Monoid to attach a log value.

Here is our new data type that specialises Writer.

case class ListWriter[A](log: List[String], value: A)

What is so interesting about this anyway? It's a monad! Importantly, its Monad implementation keeps track of the logging for us, which is important to our goal. Let's write the implementation:

val ListWriterMonad: Monad[ListWriter] = new Monad[ListWriter] {
  def flatMap[A, B](a: ListWriter[A], f: A => ListWriter[B]) = {
    val ListWriter(log, b) = f(a.value)
    ListWriter(a.log ::: log /* Monoid.append */, b)
  }
  def map[A, B](a: ListWriter[A], f: A => B) = 
    ListWriter(a.log, f(a.value))
} 

Notice in the flatMap implementation where the logged values are appended. Next we'll need some helper functions for attaching log values:

def log[A](log: String, a: A): ListWriter[A] =
  ListWriter(List(log), a)

def nolog[A](a: A): ListWriter[A] =
  ListWriter(Nil /* Monoid.empty */, a)

... now let's watch it in action. The code below is analagous to a for-comprehension. However, instead of pulling values off and naming them to the left of a <-, we flatMap values and names them to the right. We are using the explicit function calls that we defined instead of a for-comprehension:

val m = ListWriterMonad
val r = 
  m flatMap (log("computing an int", 42), (n: Int) =>
  m flatMap (log("adding 7",      7 + n), (o: Int) =>
  m flatMap (nolog(o + 3),                (p: Int) =>
  m map     (log("is even?", p % 2 == 0), (q: Boolean) =>
    !q))))
println("value: " + r.value)
println("LOG")
r.log foreach println

If you run this little snippet, you will see the final computed value and the log that was accumulated while the computation happened. Importantly, you may intercept this computation at any point and observe the current log, then continue the computation by exploiting the referentially transparent property of the expression and its sub-expressions. Note that throughout the entire computation you have not yet performed any side-effects and so you have maintained the compositional properties of the program.

You might also like to implement map and flatMap on ListWriter which will just copy the Monad implementation. I shall leave doing this for you :) This will allow you to use a for-comprehension:

val r = 
  for { 
    n <- log("computing an int", 42)
    o <- log("adding 7",      7 + n)
    p <- nolog(o + 3)
    q <- log("is even?", p % 2 == 0)
  } yield !q
println("value: " + r.value)
println("LOG")
r.log foreach println

Just like non-logging values only in a for-comprehension!

The WriterT Monad Transformer

Righto, so how do we add this logging ability to our existing for-comprehension? This is where you need the WriterT monad transformer. Again, we'll specialise it to List for logging and for the purpose of demonstration:

// The WriterT monad transformer
case class ListWriterT[M[_], A](w: M[ListWriter[A]])

This data type adds logging to values that are computed inside any value for M. It does this with its own implementation for Monad. Unfortunately, this requires partial type constructor application, which is all fine, except Scala doesn't do this very well. At least, it's a bit noisy and requires a bit of handwaving. Here it is, please bear with it:

def ListWriterTMonad[M[_]](m: Monad[M]): 
      Monad[({type λ[α]=ListWriterT[M, α]})#λ] =
  new Monad[({type λ[α]=ListWriterT[M, α]})#λ] {
    def flatMap[A, B](a: ListWriterT[M, A], f: A => ListWriterT[M, B]) =
      ListWriterT(
        m flatMap (a.w, (p: ListWriter[A]) =>
            p match { case ListWriter(log1, aa) => 
        m map     (f(aa).w, (q: ListWriter[B]) =>
            q match { case ListWriter(log2, bb) =>
        ListWriter(log1 ::: log2, bb)})
      }))
    def map[A, B](a: ListWriterT[M, A], f: A => B) = 
      ListWriterT(
        m map (a.w, (p: ListWriter[A]) =>
            p match { case ListWriter(log, aa) => 
        ListWriter(log, f(aa))
      }))
  }

The point of this monad implementation is that you can attach logging to any value M for as long as there is a Monad for M. In other words, this is how you might "add tracing within a for-comprehension." The handling of appending log values will be taken care of automatically by the Monad implementation.

For the purposes of explanation, we have deviated from how such a library would be implemented for practical use. For example, when we use the Monad implementation for ListWriterT we would probably insist on using a for-comprehension. However, we haven't directly (or indirectly) implemented flatMap or map methods on it so we cannot do this as it stands.

Nevertheless, I hope this explanation has conveyed the point of how the WriterT monad transformer solves your problem.

Now, on to a brief look at the merits and possible drawbacks of this approach.

Critique

While some of the code above may be quite abstract and even noisy, it encapsulates the algebraic concept of logging while computing a value. A library that was specifically designed to do this in a practical sense would alleviate the burden on the client code as much as possible. Coincidentally, I have implemented such a library for Scala a few years ago when I was working a commercial project.

The point of logging this way is to separate the typical side-effect (such as printing or writing to a log file) from the computation of a value with an associated log and to handle the monoidal property of logging automatically for the calling client. Ultimately, this separation leads to code that is much easier to read and reason about (believe it or not, despite some syntactic noise) and is less prone to error. Further, it assists in code reuse by combining high-level abstract functions to produce more and more specialised functions until eventually you are at the level of your specific application.

The downside to this approach is that it is not amenable to a program crash. That is, if you are, as a programmer, attempting to resolve an argument with your type-checker or runtime, then you probably want to use debugging breakpoints or print statements. Rather, the approach that I have given is more suitable for logging in production code where have assumed there to be no contradictions or bugs in your code.

Conclusion

I hope this helps!

Here is a related post on the topic.