How can HMMs be used for handwriting recognition?

Question

The problem is a bit different than traditional handwriting recognition. I have a dataset that are thousands of the following. For one drawn character, I have several sequenti

Answer 1

This problem is actually a mix of two problems:

1. recognizing one character from your data
2. recognizing a word from a (noisy) sequence of characters

A HMM is used for finding the most likely sequence of a finite number of discrete states out of noisy measurements. This is exactly problem 2, since noisy measurements of discrete states a-z,0-9 follow eachother in a sequence.

For problem 1, a HMM is useless because you aren't interested in the underlying sequence. What you want is to augment your handwritten digit with information on how you wrote it.

Personally, I would start by implementing regular state-of-the-art handwriting recognition which already is very good (with convolutional neural networks or deep learning). After that, you can add information about how it was written, for example clockwise/counterclockwise.

Answer 2

I think HMM can be used in both problems mentioned by @jens. I'm working on online handwriting too, and HMM is used in many articles. The simplest approach is like this:

1. Select a feature.
2. If selected feature is continuous convert it to discrete.
3. Choose HMM parameters: Topology and # of states.
4. Train character models using HMM. one model for each class.
5. Test using test set.

for each item:

1. the simplest feature is angle of vector which connects consecutive points. You can use more complicated features like angles of vectors obtained by Douglas & Peucker algorithm.
2. the simplest way for discretization is using Freeman codes, but clustering algorithms like k-means and GMM can be used too.
3. HMM topologies: Ergodic, Left-Right, Bakis and Linear. # of states can be obtained by trial & error. HMM parameters can be variable for each model. # of observations is determined by discretization. observation samples can be have variable length.
4. I recommend Kevin Murphy HMM toolbox.
5. Good luck.