# Exploring Metalinguistic Awareness in Pre-trained Language Models through the International Linguistics Olympiad Challenges

Junehwan Sung Hidetaka Kamigaito Taro Watanabe
Nara Institute of Science and Technology
\{sung. junehwan.sl9, kamigaito.h, taro\}@naist.ac.jp
}

\begin{abstract}
Despite significant advances in natural language processing, the degree to which these models mimic "humanlike" linguistic cognition remains uncertain. This study explores the metalinguistic awareness of Pre-trained Language Models (PLMs), focusing on their ability to comprehend the structure of language. We utilise the challenging Rosetta Stone problems from the International Linguistics Olympiad (IOL), which entail translating sentences from an unknown language solely relying on limited information. ByT5 was selected as our model due to its byte-level unbiased tokenisation and its aptness for translation tasks. Our empirical findings reveal that whilst ByT5 can learn implicit linguistic patterns without supervision, it exhibits limited metalinguistic awareness, particularly in zero-shot learning scenarios. These results highlight the need for ongoing research to enhance the depth and breadth of language understanding in PLMs and to bridge the gap towards human linguistic capabilities.
\end{abstract
## 1 Introduction

Recent advances in language models have significantly reshaped human-computer interactions across a wide range of domains. However, a pivotal question remains: To what extent do these models replicate "human-like" linguistic capabilities? This paper explores this question by investigating the metalinguistic awareness of Pre-trained Language Models (PLMs), with a particular focus on their capacity to comprehend and analyse the structure of language.

We assess this aspect of language models using the International Linguistics Olympiad (IOL) ${ }^{1)}$ challenges, specif-


ically the "Rosetta Stone" problems [1]. These challenges, fundamentally translation tasks, require participants to decipher and translate sentences from an unknown language based on limited context, making them ideal for evaluating metalinguistic awareness. We selected ByT5 [2] as our model of choice, owing to its byte-level, unbiased tokenisation that is adept at handling the diverse and lesser-known languages encountered in IOL. ByT5's proven proficiency in translation tasks is also well-aligned with the translationcentric nature of the IOL challenges.

Our experiments range from single-language tasks to more complex zero-shot scenarios, utilising IOL past papers. The findings demonstrate ByT5's ability to learn implicit linguistic patterns unsupervisedly, but also reveal its limitations in metalinguistic awareness, particularly evident in zero-shot learning contexts.

In conclusion, whilst significant progress has been made in language models, a notable gap persists in achieving human-like metalinguistic awareness. Our research provides a foundational step towards future efforts aimed at enhancing PLMs, steering them towards a deeper, more human-like understanding of language.

## 2 Related Work

The work "PuzzLing Machines: A Challenge on Learning From Small Data" by Şahin et al. [3] is a seminal study that investigates the learning capabilities and limitations of various models, from basic statistical algorithms to sophisticated architectures, using Rosetta Stone puzzles. The study illuminates the difficulties these models face in tasks requiring the understanding and manipulation of language, particularly when they are trained on small data sets. This difficulty points to a significant gap in the models' abilities to engage in deep, abstract linguistic reasoning, simi-

Preprocessing

Figure 1: Left: A Rosetta Stone problem of Inuktitut excerpted from the 2021 IOL paper ${ }^{2}$. Right: Preprocessing example, showing Inuktitut sentences converted through character mapping, retaining their English translations.

lar to the demands of the Rosetta Stone challenges in the IOL, which suggests they may focus on memorising surface patterns rather than engaging in human-like abstract reasoning.

Building upon the findings of Şahin et al., our research extends the examination of language models' abilities to a multilingual context. Whilst their study provides a benchmark for the current capabilities of various approaches in linguistic reasoning, our paper aims to further explore the metalinguistic awareness and deductive reasoning abilities in multilingual pre-trained models. By incorporating a diverse array of languages and data augmentation techniques, we delve deeper into the potential and limitations of language models, aiming to deepen our understanding of how "human-like" they are in the cognition of language.

## 3 Dataset

## 3.1 International Linguistics Olympiad

The International Linguistics Olympiad (IOL) is a prestigious competition that tests advanced linguistic skills and metalinguistic awareness - key abilities in understanding and applying language rules. In this Olympiad, the Rosetta Stone challenges [1], as shown in Figure 1, are of particular interest to our study. They feature pairs of sentences: one in a lesser-known language and its translation in the participant's native language. Participants must then translate new sentences using only these provided pairs, which tests their deep understanding of language without needing extra knowledge beyond a high school level. In case of the example in Figure 1, the solvers are asked to translate sentences in (a) and (b), using the 12 sentence pairs given.

These challenges are essential for assessing the metalinguistic abilities of Pre-trained Language Models (PLMs). By engaging PLMs with these complex translation tasks, we aim to gauge their language comprehension and their ability to mimic human cognitive processes in language analysis.

For our experiments, we have used a dataset compiled from IOL past papers dating from 2003 to 2022, focusing solely on Rosetta Stone puzzles due to their relevance in measuring metalinguistic awareness. We limited our study to translations from various source languages to English, resulting in a collection of 125 problems in 27 different languages.

## 3.2 Preprocessing

Inspired by Radford et al. [4], we adopted a concatenated data format diverging from traditional sentence pairings. Source sentences follow their English translations, separated by ' $=$ ', and distinguished from subsequent pairs by ';'. Consequently, the input takes the form ' $S_{1}=T_{1} ; S_{2}=T_{2} ; \ldots S_{n}$, where $S_{i}$ represents the $i$ th source sentence, and $T_{i}$ is its corresponding English translation. The model's task involves translating the contextually embedded source sequence $S_{n}$, with $T_{n}$ serving as the label during training. This structure, validated through extensive testing, was found to be most effective for our purposes, focusing on the model's accuracy in translating these sequences.

2) https://ioling.org/2021/

## 3.3 Data Augmentation

Recognising the proven link between dataset size and model performance [5], we devised a data augmentation strategy tailored for the distinctive challenges of IOL problems. Our approach entails creating permutations of each language's unique character set and pairing these permuted sequences with their original translations to form new sentence pairs that are structurally distinct yet retain the linguistic features of the original. The process is as follows:

1. Identifying unique characters: Catalogue the unique characters from the language, ordering them in ascending sequence.

2. Generating permutations: Implement a derangement algorithm that rearranges the characters ensuring none remain in their original position, thereby producing a comprehensive set of permutations that grow exponentially with the number of characters.

3. Augmenting data: Combine the original character list identified in step 1 with its deranged counterpart. Subsequently, transform the source sentences using this new mapping to foster a dataset that exposes the model to diverse linguistic structures whilst preserving syntactic coherence.

For visual reference, please see Figure 1. This augmentation technique generates a novel linguistic dataset that preserves both the solvability of translation tasks and the authenticity of the original linguistic features, thereby constructing a platform to simulate metalinguistic awareness in our models. The training dataset used in the following experiments composed exclusively of these augmented sequences, deliberately excluding any unmodified original text, to test the model's ability to infer linguistic rules from purely metalinguistic signals. We varied the augmentation size from 1 to 2,000 to rigorously assess this capability.

## 4 Experiment

To examine metalinguistic awareness in language models, we conducted a series of experiments using the ByT5 model [2]. Chosen for its byte-level tokenisation and proficiency in translation tasks, ByT5 is well-suited for the IOL challenges dealing with translations in extremely minor languages. Each experiment was replicated five times with a fixed seed for consistency. We employed BLEU-2 [6] as our metric, given the brevity of our dataset's sentences. The following sections will outline our methodologies and findings, highlighting how the models process and understand language structures in a manner similar to human cognition. This research primarily explores translation into English, setting aside reverse translation for future work. For detailed results, please refer to Table 1 and Appendix A.

## 4.1 Single Language Experiment

In our initial experiment, we adopted the simplest setup by focusing on a single language. Madak was selected due to its extensive set of questions, providing the largest dataset for translation from English. The training set consisted exclusively of augmented synthetic sequences derived from Madak's original texts, whilst the originals were deliberately omitted. The validation and test sets, however, included Madak's questions in their original, unaugmented form. This setting enables us to emulate a multilingual environment where each language reflects the unique features of the Madak language.

The results exhibit a general upward trend, confirming the model's ability to learn and analyse implicit linguistic patterns unsupervisedly. Despite a sudden drop in BLEU-2 scores at an augmentation size of 200 , there is a consistent improvement as the augmentation size increases. Notably, the scores steadily rise until they reach a peak at an augmentation size of 1,000 . This ascending pattern underscores the model's growing adeptness at deciphering Madak's language structure, demonstrating the solvability of the tasks and the efficacy of our data augmentation strategy.

## 4.2 Multilingual Adaptability with Added Languages

To evaluate the model's learning ability with diverse linguistic features, we expanded our experimental setup by introducing additional languages alongside Madak. The training split now comprises augmented sequences ${ }^{3}$, from both Madak and another language, whereas the validation and test splits retained Madak's original questions, consistent with our initial experiment design.

For this expanded approach, we selected two distinct languages: Kilivila, sharing the same language family as Madak, and Inuktitut, from a different language family. This choice was strategic, aimed at exploring the impact of

3) the un-augmented original sentences were excluded from the set

Table 1: BLEU-2 scores of the test split for each experiment

linguistic diversity on the model's learning capabilities.

When Kilivila was added, the BLEU-2 scores showed a marked improvement even with smaller augmentation, compared with the Madak-only scenario. The addition of Inuktitut further accentuated this trend, indicating not only the model's adaptability to linguistic variety but also its enhanced translation capabilities. These results suggest that incorporating diverse linguistic inputs can significantly bolster a model's learning process, highlighting the benefits of multilingual training environments for developing more robust language models.

## 4.3 Zero-Shot Learning and metalinguistic Awareness

In our final experiment, we established a zero-shot learning environment, where the training set with augmented sequences from all 26 languages except Madak, deliberately excluding the un-augmented data. Conversely, the validation and test sets comprise solely the original, unaugmented Madak sequences, aligning with the methodology of prior experiments. Such a design is to assess whether the model could extrapolate its learnt structures to an unseen language, thereby demonstrating metalinguistic awareness. This approach contrasted with earlier experiments, focusing exclusively on augmentation sizes of 1,000 and 2,000 - identified as the data volumes where BLEU-2 scores were most optimal in prior trials.

Although we observed a slight upward trend in BLEU-2 scores as the augmentation size increased from 1,000 to 2,000 in this zero-shot scenario, the overall scores remained low, peaking at just around 3. This suggests that whilst there is a marginal improvement with increased data, the model's performance did not reach a level that could be characterised as demonstrating metalinguistic awareness. This plateau in performance, even with substantial data augmentation, points to the limitations of the current model in adapting to new languages without direct training. The modest increase in scores with larger augmentation sizes, however, offers a glimmer of hope that with even more data or perhaps a refined approach to training, the models might begin to show signs of the desired metalinguistic capabilities.

## 5 Conclusion

The initial experiment with the Madak language has successfully verified that models are capable of learning language structures with increased data augmentation in an unsupervised manner. This was particularly evident when the data augmentation size reached 1,000 , where we saw the models' performance peak, affirming the effectiveness of our approach.

In the subsequent experiment, the introduction of languages from both the same and different families as Madak resulted in consistent performance improvements. This was especially pronounced with the addition of Inuktitut, suggesting that linguistic diversity is potentially beneficial to model learning and adaptability.

Finally, the zero-shot experiment tested the models' ability to apply learnt knowledge to an unseen language. Although there was a marginal improvement in performance with larger augmentation sizes, the models did not demonstrate a strong metalinguistic awareness, as indicated by the BLEU-2 scores capped at 3.

These findings collectively point to a nuanced understanding of language models' current state. Whilst promising strides have been made, particularly in terms of handling linguistic diversity, there is still a considerable gap in achieving genuine metalinguistic awareness. Future research, potentially involving more extensive data augmentation or new training methodologies, can be considered to bridge this gap. Our study lays a foundational step towards such future explorations, aiming to empower language models with human-like linguistic abilities.

## 6 Acknowledgement

We wish to express our heartfelt gratitude to all those who contributed to the success of this research. We are also immensely grateful to the organisers of the International Linguistics Olympiad for providing the datasets and problems that formed the basis of our study.

## References

[1] Bozhidar Bozhanov and Ivan Derzhanski. Rosetta stone linguistic problems. In Ivan Derzhanski and Dragomir Radev, editors, Proceedings of the Fourth Workshop on Teaching NLP and CL, pp. 1-8, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.

[2] Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. CoRR, Vol. abs/2105.13626, , 2021.

[3] Gözde Gül Şahin, Yova Kementchedjhieva, Phillip Rust, and Iryna Gurevych. PuzzLing Machines: A Challenge on Learning From Small Data. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1241-1254, Online, July 2020. Association for Computational Linguistics.

[4] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

[5] Houman Mehrafarin, Sara Rajaee, and Mohammad Taher Pilehvar. On the importance of data size in probing fine-tuned models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pp. 228-238, Dublin, Ireland, May 2022. Association for Computational Linguistics.

[6] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, p. 311-318, USA, 2002. Association for Computational Linguistics.

## A Appendix

Augmented Madak (Train); Madak (Valid, Test)

Figure 2: Experiment 1 - Single Language (Madak)

Madak + Kilivila (Train); Madak (Valid, Test)
- valid = test " " madak_only_valid " " madak_only_test

(a) Madak + Kilivila (same family)
Madak + Inuktitut (Train); Madak (Valid, Test)

(b) Madak + Inuktitut (different family)

Figure 3: Experiment 2 - Madak + Another Language (compared with Madak-only)

Madak (7 questions)

Data augmentation size

Figure 4: Experiment 3 - Zero-shot