State-of-the-art handwritten text recognition models make frequent use of deep neural networks, with recurrent and connectionist temporal classification layers, which perform recognition over sequences of characters. This architecture may lead to the model learning statistical linguistic features of the training corpus, over and above graphic features. This in turn could lead to degraded performance if the evaluation dataset language differs from the training corpus language. We present a fundamental study aiming to understand the inner workings of OCR models and further our understanding of the use of RNNs as decoders. We examine a real-world example of two graphically similar medieval documents but in different languages: rabbinical Hebrew and Judeo-Arabic. We analyze, computationally and linguistically, the cross-language performance of the models over these documents, so as to gain some insight into the implicit language knowledge the models may have acquired. We find that the implicit language model impacts the final word error by around 10%. A combined qualitative and quantitative analysis allow us to isolate manifest linguistic hallucinations. However, we show that leveraging a pretrained (Hebrew, in our case) model allows one to boost the OCR accuracy for a resource-scarce language (such as Judeo-Arabic). All our data, code, and models are openly available at https://github.com/anutkk/ilmja.
|Title of host publication||Document Analysis and Recognition – ICDAR 2023 - 17th International Conference, Proceedings|
|Editors||Gernot A. Fink, Rajiv Jain, Koichi Kise, Richard Zanibbi|
|Publisher||Springer Science and Business Media Deutschland GmbH|
|Number of pages||18|
|State||Published - 2023|
|Event||17th International Conference on Document Analysis and Recognition, ICDAR 2023 - San José, United States|
Duration: 21 Aug 2023 → 26 Aug 2023
|Name||Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|
|Conference||17th International Conference on Document Analysis and Recognition, ICDAR 2023|
|Period||21/08/23 → 26/08/23|
Bibliographical noteFunding Information:
Supported in part by the Deutsch Foundation, the Israeli Ministry of Science and Technology (grant number 3-17516) and a grant from Tel Aviv University Center for Al and Data Science (TAD) in collaboration with Google, as part of the initiative “AI and DS for Social Good”.
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.
- Handwritten text recognition
- Hebrew manuscripts
- Language model
- Optical character recognition
- Transfer learning
ASJC Scopus subject areas
- Theoretical Computer Science
- Computer Science (all)