Is there a tool (or an algorithm) to convert a finite state machine into a regular expression?
(not the other way around, that would be easy).
There are several algorithms to perform this task: the "state-elimination method" from Brzozowski and Mc Cluskey, the resolution of a system of linear equation, the method from McNaughton and Yamada, etc. They are very well described in Automata and rational expressions by Jacques Sakarovitch.
The state-elimination method in particular is simple to understand. The key idea is that you are going to build an automaton labeled by rational (aka regular) expressions rather than letters. First, make sure you have a single initial state and a single final state (you may add fresh states and spontaneous transitions if necessary). Then choose a state s to eliminate, say state 1 in the following picture.
Then consider all the couples (p, q) where p is a predecessor (states from which a transition reaches s, 0 in the picture) and q a successor (state 2). For each such couple (p, q) add a transition from p to q which is labeled by E(p, q) + E(p, s)E(s, s)*E(s, q) where E(p, s) means "the expression that labels the transition from p to s. Once you treated all the couple (p, q), remove the state s. In the previous example:
Do that until you eliminated all the inner states (i.e., keep the initial state and the final state), and just read the result on the transition from the initial state to the final state (here d+ab*c).
You may toy with this algorithm using Vcsn, a tool for rational expressions and automata. Here is a complete example you may reproduce at Vcsn Sandbox.