# [B23] KuroNet: End-to-end Kuzushiji Transcription System for Understanding Historical Documents

Tarin Clanuwat $^{11}$, Alex (amb ${ }^{2)}$, Asanobu Kitamoto1)

1) ROIS-DS Center for Open Data in the Humanities, National Institute of Informatics

2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430

2) Mila, Quebec Artificial Intelligence Institute, Université de Montréal

E-mail: tarin@nii.ac.jp
## 【Abstract】

Kuzushiji was used as the dominant writing system in Japan for over a thousand years. However, following a reform of the Japanese language in 1900, Kuzushiji was removed from the regular school curriculum. As a result, most Japanese people today cannot read Kuzushiji. This causes difficulty in archiving and examining Kuzushiji documents. KuroNet is an end-toend Kuzushiji transcription system that transcribes Kuzushiji documents and takes one second per page. A service for running KuroNet is available online through IIIF Curation Viewer developed by the ROIS-DS Center for Open Data in the Humanities. In this paper, we explain the basic idea of KuroNet, such as the architecture and the capabilities of KuroNet. We also discuss how KuroNet can help museums and libraries preserve historical documents, making Kuzushiji documents more accessible, and how people can learn about Japanese culture and history from using KuroNet. Finally, we suggest directions for future work which could enhance the usage of Kuzushiji documents such as full-text search, text-to-speech, and other visualization projects.

## 1. Introduction

Kuzushiji or cursive style Japanese characters were used in the Japanese writing and printing system for over a thousand years. However, the standardization of Japanese language textbooks known as the Elementary School Order in 1900 unified the writing type of Hiragana. This made the Kuzushiji writing style obsolete from school curricula and incompatible with modern printing systems. Therefore, most Japanese natives cannot read books written just 150 years ago. This is the main reason why despite the ongoing efforts to create digital copies of these documents, most of the knowledge, history, and culture contained within these texts remain inaccessible to the general public. This has led to a great deal of interest in automatically converting these documents into modern Japanese characters. Our Kuzushiji recognition system, which we call KuroNet [1] achieves superior results in machine-automated Kuzushiji transcription. Even though there remains room for further improvement, KuroNet makes Kuzushiji documents more accessible to the general public. This can help to increase efficiency in archiving and preserving historical documents.

## 2. KuroNet

## 2. 1 How KuroNet Works

CODH released a large-scale Kuzushiji dataset (http://codh.rois.ac.jp/char-shape/) created by the National Institute of Japanese Literature. As of November 2019, this dataset includes 1,086,326 character images of 4,328 character types cropped from 44 classical books. The KuroNet method is motivated by the idea of processing an entire page of text together, with the goal of capturing both long-range and local dependencies. KuroNet uses object detection algorithms (in our case is U-net [2]) to capture the locations of characters then classify them into modern Japanese character Unicodes. KuroNet begins with a training process, in which the KuroNet model is trained with all 44 books from the Kuzushiji dataset. It takes about 5 days with one NVidia Titan RTX GPU. At test time, we evaluate by running the U-Net on the entire uncropped page but at a higher resolution than was used for training (which is possible because the U-Net is a fully
convolutional architecture). As a result, KuroNet can transcribe one Kuzushiji page within only 1.5 seconds using a single Titan RTX GPU. Since KuroNet doesn't assume the characters have a sequential layout, the system is robust in transcribing Kuzushiji page with a complex layout. However, the transcription result relies greatly on the quality of the image and size of characters on the page.

Figure 1 Transcription result from KuroNet

## 2. 2 The Capabilities of KuroNet

Since KuroNet is trained on a specific Kuzushiji dataset, therefore, what KuroNet can transcribe with high performance is limited by the scope of the dataset. The Kuzushiji dataset was created mostly from woodblock-printed books from the Edo period. Only 3 out of the 44 books are handwritten. The difference between handwritten and printed books is the readability of the characters. The purpose of printed books is to be read by a lot of people. However, in personal documents such as diaries, records or letters, the variation in how characters are written is much greater. In a real-world situation, the number of handwritten documents is far beyond the number of printed books. This is why there is a lot of demand for transcribing handwritten Kuzushiji documents which have survived all over Japan. Many of them come from old storehouses that belonged to individual owners. If the documents are found in bad conditions (mold or moisture), and thus might be thrown away before examining their content. Even though KuroNet performs very well with high accuracy on printed books from the Edo period, it will struggle when transcribing handwritten documents. The way to solve this problem is by gathering more data from handwritten documents, and retraining with this additional data included. In the Kuzushiji dataset, we have to specify the bounding box of each character by human which is both a budget and time-consuming process. However, we can now use KuroNet to estimate the bounding box from Kuzushiji page (as shown in figure 2) which will help accelerate the dataset creation process.

Another issue with KuroNet is that the time period that documents were created is also critical in transcription. Some characters appeared in the older era might not be seen much in the Edo period such as many Hiragana characters in earlier periods don't appear often in documents in the Edo period. This is why we will need a lot more character data to make KuroNet being able to transcribe those handwritten documents and expanding the scope of the documents outside the Edo period. Using KuroNet to provide bounding boxes along with labels for simple characters while allowing humans to provide labels for challenging characters could potentially help to create additional data more quickly and efficiently.

## 3. KuroNet for Museums and Libraries

KuroNet will be released as open-source software in the future. However, KuroNet software itself is quite hard to use if users don't have the right technical skills. For example, users have to have the ability to install Pytorch, install CUDA, and run python code. Additionally, the computer should have a reasonably powerful GPU. For
this reason we decided to release a KuroNet web service first so that anyone can use it without having any particular technical skills. We made the Kuzushiji recognition web service available from IIIF Curation Viewer which was developed by CODH. The International Image Interoperability Framework (IIIF) is the universal standard for describing and delivering images over the web and is the result of collaborative efforts across universities, museums, libraries, and other cultural heritage institutions around the world. At the moment, if a document has IIIF manifest URI, it can be transcribed with our KuroNet web service. One downside of our current KuroNet web service is we still don't allow users to upload their Kuzushiji document images to the server due to privacy concerns and technical issues.

The advantage of using KuroNet to transcribe documents for libraries and museums is that if the documents are unknown, librarians or curators can use KuroNet to transcribe documents and use extracted keywords and phrases for categorization. Note that this uses aggregate properties of a text, so it does not require the model to be perfectly accurate. This may help accelerate many processes in indexing or archiving. In museums, curators can also use KuroNet to transcribe documents and then edit the result for perfect accuracy to prepare the transcription for public exhibition. However, using KuroNet alone to do transcription for museum exhibition is not practical since the transcription must be $100 \%$ accurate for this task.

## 4. KuroNet for Education

Since KuroNet can help to make historical documents more accessible, anyone who is interested in learning more about Kuzushiji or anything regarding Kuzushiji documents such as history or literature will directly benefit from it. Until now, reading Kuzushiji documents has been exclusively for trained professionals which has made some people lose interest in them. Successfully preserving historical documents requires the support of the general public. Being able to read Kuzushiji documents may make people appreciate them more and understand why preservation is needed. Moreover, we can also make people be more familiar with Kuzushiji documents. For example, in classical literature class, instead of using transcribed text in a textbook, students could try to read from the original Kuzushiji text from KuroNet transcription. This is possible because KuroNet predicts characters along with their positions on the original page in the text, and thus the original text (including illustrations and layout) can be read with modern text overlayed. In histories or classical literature research, researchers spend a lot of time finding specific information from Kuzushiji documents by reading them one page at a time. This can be accelerated by using KuroNet to search for specific keywords and phrases, and then allowing a researcher to manually inspect the most relevant parts of documents. A salient property of the search task is that it is robust to some amount of error rate, and can still be useful for saving humans time even if it is not perfectly accurate.

Figure 2 Bounding boxes generated from KuroNet

## 5. Future Work

There are many things that can be done to improve KuroNet. For example, if we can get feedback from human users in order to get correct labels for image, we can get more information about where KuroNet performs poorly and add more data for training.

From July to October 2019, CODH hosted Kaggle Kuzushiji Recognition Competition [3] which gathered machine learning researchers and engineers from around the whole to help us build the Kuzushiji transcription system. We plan to use the software and idea [4] we got from the competition to improve KuroNet's accuracy. We have also released the KuroNet paper to the public, and we hope to see more research from the community on how this system can be improved.

One of the biggest challenges for future work is to get sequential text output from KuroNet. This is challenging because of the irregular layout on some pages. We are currently exploring the task of automatically converting these character coordinate lists into a single text sequence for a page using a learned reading order. These text sequences are necessary for machine translation, cataloging, and search, which are critical for Kuzushiji recognition research. From the text sequence, we can also try creative projects such as text-tospeech or interesting visualizations to make Kuzushiji documents more interesting.

One critical milestone in Kuzushiji recognition research is to create a full-text search system for Japanese historical documents. We can use KuroNet to transcribe a large amount of image data and perform searches for given keywords and phrases in the text (which requires both a knowledge of the spatial position as well as the ordering of characters on a page). This will help make research faster and may lead to new discoveries in Japanese history.

## 6. Conclusion

KuroNet is a Kuzushiji recognition system that can help make historical documents more accessible. Museums, libraries and archives can benefit from it both by having an automatic search and discovery capability, as well as producing preliminary transcriptions for public exhibition. While KuroNet works well on the type of data it was trained on, it may perform worse on other types of data not seen during training. We hope to setup partially-automated pipelines to efficiently add more labeled data, expanding the domains where KuroNet performs well. We also believe that the support of the public is necessary for historical preservation. We hope that KuroNet, by providing transcriptions which are overlaid on the original (potentially illustrated) text, will help to make Kuzushiji documents more popular with the general public. As a result, we hope that this will help to grow a popular movement to preserve the cultural heritage of Japan.

## References

[1] Tarin Clanuwat, Alex Lamb, Asanobu Kitamoto, "KuroNet: Pre-Modern Japanese Kuzushiji Character Recognition with Deep Learning", 15th International Conference on Document Analysis and Recognition (ICDAR2019), arXiv:1910.09433, 2019.

[2]Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer.

[3] Asanobu KITAMOTO et al. "Progress and Results of Kaggle Machine Learning Competition for Kuzushiji Recognition", Proceedings of IPSJ SIG Computers and the Humanities Symposium 2019, pp. 223-230, 2019.

[4] Kaggle Kuzushiji Recognition Competition, https://www.kaggle.com/c/kuzushiji-recognition

この記事の著作権は著者に属します。この記事は Creative Commons 4.0 に基づきライセンスされます(http://creativecommons.org/licenses/by/4.0/)。出典を表示することを主な条件とし、複製、改変はもちろん、営利目的での二次利用も許可されています。