DEFORMABLE CONVOLUTION FOR NON-HORIZONTAL HANDWRITING RECOGNITION

Abstract

Offline Handwritten Text Recognition (OHTR) aims to convert handwriting images into text using machine learning algorithms. Handwriting poses a challenge due to the variability in character shapes and orientations. Thus, learning robust features is a critical step in improving the generalization of classifiers. Traditionally, convolutional neural networks (CNNs) have been employed for this purpose, either for feature extraction or as part of fully convolutional architectures. Convolutional layers use kernels to sample all pixels and encode them into representative features. However, the standard sampling process does not account for the varying nature of handwriting, particularly for handwriting that exhibits a wavy skew. In contrast, deformable convolutional layers, which have been recently proposed, allow the model to learn relevant pixel sampling locations, which can make them especially suited for handling non-horizontal handwriting. In this study, we assessed the robustness and generalization performance of deformable convolutions compared to standard convolutions to evaluate their suitability for offline handwriting text recognition (OHTR) tasks involving non-horizontal handwriting. Experiments conducted with fully convolutional neural network baseline on the IAM and ICDAR 2017 datasets, using augmented samples of non-horizontal handwriting, demonstrate that deformable convolutions show promise in classifying wavy handwriting, even when only a few deformable layers are used. These improved the baseline by 1% on the augmented IAM dataset. However, ablation studies suggest that further optimization is needed, particularly in depth-wise settings, to fully unlock their potential. Additionally, the limited availability of real-world handwriting datasets with diverse orientations restricts comprehensive evaluation of these models.