Directed probability graph - algorithm to reduce cycles?

Question

Consider a directed graph which is traversed from first node 1 to some final nodes (which have no more outgoing edges). Each edge in the graph has a probability ass

Answer 1

I'm not expert in the area of Markov chains, and although I think it's likely that algorithms are known for the kind of problem you present, I'm having difficulty finding them.

If no help comes from that direction, then you can consider rolling your own. I see at least two different approaches here:

1. Simulation.

Examine how the state of the system evolves over time by starting with the system in state 1 at 100% probability, and performing many iterations in which you apply your transition probabilities to compute the probabilities of the state obtained after taking a step. If at least one final ("absorbing") node can be reached (at non-zero probability) from every node, then over enough steps, the probability that the system is in anything other than a final state will decrease asymptotically toward zero. You can estimate the probability that the system ends in final state S as the probability that it is in state S after n steps, with an upper bound on the error in that estimate given by the probability that the system is in a non-final state after n steps.

As a practical matter, this is the same is computing Trⁿ, where Tr is your transition probability matrix, augmented with self-edges at 100% probability for all the final states.

2. Exact computation.

Consider a graph, G, such as you describe. Given two vertices i and f, such that there is at least one path from i to f, and f has no outgoing edges other than self-edges, we can partition the paths from i to f into classes characterized by the number of times they revisit i prior to reaching f. There may be an infinite number of such classes, which I will designate C_if(n), where n represents the number of times the paths in C_if(n) revisit node i. In particular, C_ii(0) contains all the simple loops in G that contain i (clarification: as well as other paths).

The total probability of ending at node f given that the system traverses graph G starting at node i is given by

Pr(f|i, G) = Pr(C_if(0)|G) + Pr(C_if(1)|G) + Pr(C_if(2)|G) ...

Now observe that if n > 0 then each path in C_if(n) has the form of a union of two paths c and t, where c belongs to C_ii(n-1) and t belongs to C_if(0). That is, c is a path that starts at node i and ends at node i, passing through i n-1 times between, and t is a path from i to f that does not pass through i again. We can use that to rewrite our probability formula:

Pr(f|i,G) = Pr(C_if(0)|G) + Pr(C_ii(0)|G) * Pr(C_if(0)|G) + Pr(C_ii(1)|G) * Pr(C_if(0)|G) + ...

But note that every path in C_ii(n) is a composition of n+1 paths belonging to C_ii(0). It follows that Pr(C_ii(n)|G) = Pr(C_ii(0)|G)ⁿ⁺¹, so we get

Pr(f|i) = Pr(C_if(0)|G) + Pr(C_ii(0)|G) * Pr(C_if(0)|G) + Pr(C_ii(0)|G)² * Pr(C_if(0)|G) + ...

And now, a little algebra gives us

Pr(f|i,G) - Pr(C_if(0)|G) = Pr(C_ii(0)|G) * Pr(f|i,G)

, which we can solve for Pr(f|i,G) to get

Pr(f|i,G) = Pr(C_if(0)|G) / (1 - Pr(C_ii(0)|G))

We've thus reduced the problem to one in terms of paths that do not return to the starting node, except possibly as their end node. These do not preclude paths that have loops that don't include the starting node, but we can we nevertheless rewrite this problem in terms of several instances of the original problem, computed on a subgraph of the original graph.

In particular, let S(i, G) be the set of successors of vertex i in graph G -- that is, the set of vertices s such that there is an edge from i to s in G, and let X(G,i) be the subgraph of G formed by removing all edges that start at i. Furthermore, let p_is be the probability associated with edge (i, s) in G.

Pr(C_if(0)|G) = Sum over s in S(i, G) of p_is * Pr(f|s,X(G,i))

In other words, the probability of reaching f from i through G without revisiting i in between is the sum over all successors of i of the product of the probability of reaching s from i in one step with the probability of reaching f from s through G without traversing any edges outbound from i. That applies for all f in G, including i.

Now observe that S(i, G) and all the p_is are knowns, and that the problem of computing Pr(f|s,X(G,i)) is a new, strictly smaller instance of the original problem. Thus, this computation can be performed recursively, and such a recursion is guaranteed to terminate. It may nevertheless take a long time if your graph is complex, and it looks like a naive implementation of this recursive approach would scale exponentially in the number of nodes. There are ways you could speed the computation in exchange for higher memory usage (i.e. memoization).

---

There are likely other possibilities as well. For example, I'm suspicious that there may be a bottom-up dynamic programming approach to a solution, but I haven't been able to convince myself that loops in the graph don't present an insurmountable problem there.