A HYBRID LARGE VOCABULARY HANDWRITTEN WORD RECOGNITION SYSTEM USING NEURAL NETWORKS WITH HIDDEN MARKOV MODELS

Introduction

During the last few years, HMMs have become a very popular approach in handwriting recognition. One of the reasons is their higher performance in medium to large vocabulary applications where segmentation recognition methods are used to cope with the difficulties of segmenting words into characters. Segmentation–recognition methods first loosely segment (oversegment) words into graphemes that ideally consist of either characters or parts of characters, and use dynamic programming techniques together with a lexicon to find the definitive segmentation as well as the best word hypotheses. Many systems use HMMs to model sub–word units (characters) and the Viterbi algorithm to find the best match between a sequence of observations and the models. 

The Viterbi algorithm is optimal in the sense of maximum likelihood and it looks at the match of the whole sequence of features (observations) before deciding on the most likely state sequence. This is particularly valuable in applications such as handwritten word recognition where an intermediate character may be garbled or lost, but the overall sense of the word may be detectable. On the other hand, the local information is somewhat overlooked. Furthermore, the conditional independence imposed by the Markov Model (each observation is independent of its neighbors) prevents an HMM from taking full advantage of the correlation that exists among the observations of a single character. Neural network classifiers exhibit powerful discriminative properties and they have been used in handwriting recognition particularly with digits, isolated characters, and words in small vocabularies . 

However, the use of NNs in the recognition of handwritten words from larger vocabularies depends heavily on a very efficient segmentation scheme. Due to the lack of such an efficient segmentation scheme, NNs are usually employed in combination with other classifiers, e.g. hybrid NN/HMM approaches that use NNs to estimate a priori probabilities, or that use NNs to validate grapheme hypotheses generated by HMM classifiers. 

In this project an approach to integrate NNs and HMMs in a probabilistic framework that takes advantage of the good properties of both methods: the generation of an N–best list of word hypotheses by the HMM classifier together with the segmentation of each hypothesis into characters and the character modeling properties of the NN classifier. The NN classifier uses the segmentation information provided by the HMM classifier to go back to the input image and extract new features more suitable for isolated character recognition. The NN classifier scores the segments of each N–best word hypothesis and such scores are further combined with the scores generated by the HMM classifier. Finally, the N–best list is reordered according to the new composite scores, shifting up the correct word hypothesis.