# Compositional augmentation policy using different formulas for the notion of middle sentence for low resource machine translation

CHEN Jin and Yves LEPAGE
早稲田大学大学院情報生産システム研究科
jin.chen@asagi.waseda.jp yves. lepage@waseda.jp
}

\begin{abstract}
Middle sentence generation is a technique that, given a start sentence and an end sentence, outputs a sentence with middle semantics. We aim to further explore formulas for middle sentence generation and construct a compositional policy to combine these formulas into better translation models. We study particular and general types of formulas: particular formulas are based on specific words and general formulas are based on the IDF score of each individual word. We use these formulas as corpus augmentation operations and define a policy that automatically constructs a breadth-first tree which finds the node with the least perplexity as the best node. Results show that our policy provides significant improvements over our baseline.
\end{abstract
## 1 Introduction

The idea of middle sentence generation is based on a specific analogy [1]:

## Start : Middle :: Middle : End

It takes two sentences from the corpus as start sentence and end sentences and generates a new sentence whose semantics lies at the semantic midpoint of the two start and end sentences. If sentences are represented by vectors, the basic formula for the vector of a middle sentence is thus as follows:

$
\overrightarrow{\text { Middle }}_{\text {basic }}=\frac{1}{2} \times(\overrightarrow{\text { Start }}+\overrightarrow{\text { End }})
$

Some studies have demonstrated the effectiveness of data augmentation methods based on the idea of middle sentence generation for natural language processing tasks $[2,3,4,5,6]$. However, they have limitations as they explore various different formulas by evaluating various data augmentation results that use these different formulas.

This paper aims to continue the exploration of genera- tion formulas that use the notion of middle sentence, by constructing a comprehensive framework that builds upon several proposed generation formulas.

## 2 Sentence generation based on the notion of middle sentence

## 2.1 Basic and renormalized formulas

New sentences can be generated directly as middle sentences. This has been explored in works like [4]. The main drawback is that the middle vector might not be a vector that actually fits a sentence vector. The consequence is that, when decoding from the vector, the generated sentence suffers problems: repeated words, sentences somehow falling short off, inconsistent semantics.

So as to remedy to this problem and release the constraint on the middle vector, more freedom can be given to generation by creating a vector sustained by the start and the middle sentence, instead of the start and end sentence [6]. This allows the creation of sentences with possibly new semantics

$
\overrightarrow{\text { End }}_{\text {basic }}=2 \times \overrightarrow{\text { Middle }}-\overrightarrow{\text { Start }}
$

The previous formulas (1) and (2) do not take into account the fact that the norms of the generated middle or end vectors might be too short for being an accurate vector representation of a sentence. Renormalization is an answer to this problem. It makes the vectors easier to decode into reasonable sentences by reducing the non-felicitous influence of too short norms $[4,5,6]$. The renormalization formulas are:

$
\text { Middle }_{\text {renorm }}=\frac{\| \overrightarrow{\text { Start }\|+\| \overrightarrow{\text { End }} \|}}{\|\overrightarrow{\text { Start }}+\overrightarrow{\text { End }}\|} \times \overrightarrow{\text { Middle }_{\text {basic }}}
$

$
\overrightarrow{\text { End }}_{\text {renorm }}=\frac{2 \times \| \text { Middle }\|-\| \overrightarrow{\text { Start }} \|}{\| 2 \times \text { Middle }- \text { Start } \|} \times \overrightarrow{\text { End }}_{\text {basic }}
$

In this paper, we adopt generation of the end sentence.

## 2.2 Influence of specific words

Taking into account low-frequency words can provide richer and more acute semantics to the generated vector representations. For the same reasons, giving some weight to hapaxes can lead to vectors positively biased towards rarer formulations. By contrast, inhibiting the influence of more frequent words can solve the problem of overloading the generated sentence with grammatical words. For all the previous reasons, we propose a series of formulas that leverage the frequency of different words.

## 2.2.1 Low frequency words

It has been observed that giving more weight to rare words can enrich the calculated sentence vectors and can lead to better generated sentences [4]. The low frequency words in appearing in the two put sentences can be added to the sentence vector weighted by some factor $\lambda$. Although the original proposal was for middle sentences, we apply it to end sentences:

$
\overrightarrow{\text { End }_{\text {lowfreq }}}=\overrightarrow{\text { End }_{\text {renorm }}}+\lambda \times \overrightarrow{w o r d}_{\text {lowfreq }}
$

## 2.2.2 Hapaxes

A reinforced view over the previous one that took into account low frequency words, is to consider the addition of the average of all hapax embeddings in the start and middle sentences.

$
\overrightarrow{\text { End }}_{\text {hapaxes }}=\frac{1}{1+n} \times\left(\overrightarrow{\text { End }}_{\text {basic }}+\lambda \times \sum_{1}^{n} \overrightarrow{\text { word }}_{\text {hapaxes }}\right)
$

## 2.2.3 Stop words

The previous formula for hapaxes added information to the sentence vector. Similarly, and by opposition, one can imagine subtracting the average vector of all embeddings of stop words to reduce their influence on the generated sentence.

$
\overrightarrow{\operatorname{End}}_{\text {stopwords }}=\overrightarrow{\operatorname{End}}_{\text {renorm }}-\left(\lambda \times \frac{1}{n} \sum_{1}^{n} \overrightarrow{\text { word }}_{\text {stopwords }}\right)
$

## 2.3 Generalization: taking idf of each word into account

A generalization of the previous formula can be obntained by taking into account their inverse document frequecy (idf). Here, documents are each individual sentence. This allows to balance the information of words according to tehir frequency. The additional part is adapted through the factor $\lambda$.

$
\overrightarrow{E n d}_{\text {all }}=\overrightarrow{\operatorname{End}}_{\text {renorm }}-\left(\lambda \times \frac{1}{n} \sum_{1}^{n}(\text { idf } \times \overrightarrow{\text { word }})\right)
$

In order to not influence too much by the norm of each word, this last formula renormalizes words before incorporating their embedding to the calculated vector.

$
\overrightarrow{\text { End }}_{\text {all+norm }}=\overrightarrow{\text { End }}_{\text {renorm }}-\left(\lambda \times \frac{1}{n} \sum_{1}^{n}\left(\frac{\text { idf }}{\| \text { word } \|} \times \overrightarrow{\text { word }}\right)\right)
$

## 3 Compositional augmentation policy

Text AutoAugment is a compositional framework whose core idea is to generate new synthetic text by combining individual editing operations to form a complete sequence [7]. The framework is not only capable of generating multiple sequence instances at once, but it can also automatically select the best-performing augmented dataset among all the ones generated. Text AutoAugment has been proposed for text classification.

We propose to adopt this compositional framework for data augmentation for the task of data augmentation in machine translation. The operations in our framwork will be the proposed formulas for sentence generation. Our framework defines a similar policy to Text AutoAugment, called $\mathscr{P}$, that contains $N$ operations $\mathscr{O}$. For each operation 0 , we define:

$
0=\langle t, \lambda, \theta\rangle
$

where:
1. $t$ is the type of generation formula, i.e. the five proposed formulas and the renormalized end sentence formula.
2. $\lambda$ is the weight of the extra term in the formula. Note that the renormalized end sentence formula can be seen as the multiplication of $\lambda$ with a zero vector. $\lambda \in[0,1]$.
3. $\theta$ represents the threshold for sentence filtration. We use Euclidean distance to determine whether the bilingual sentences are parallel. $\theta \in[0.3,0.6]$.

Our framework takes a parallel corpus as input, stores it to the root node of an exploration tree, and trains a machine translation model without any augmented data from the original parallel corpus. Subsequently, for each node, the framework creates $N$ daughter nodes according to the number of operations in the policy $\mathscr{P}$. The corpus on a daughter node is obtained by data augmentation using the corresponding operation 0 .

At a tree depth of 1, the node's model is trained on the corresponding corpus while at depths greater than 1, the node's model is fine-tuned on the parent node's model using its own corpus. When the child node's perplexity is less than the parent node's, the end sentence generation of daughter nodes stops and this daughter node is discarded.

The framework will finally compare perplexities of all models on the leaves and select the node with the smallest perplexity as the best node, with the path from the root node to that node. This path represents the best operational solution. The complete framework is illustrated in Figure 1.

Figure 1 Proposal: compositional augmentation framework

## 4 Experimental setup

## 4.1 Machine translation engine

We use the OpenNMT-py toolkit [8] to create all the necessary models. Each model is built using an encoderdecoder architecture based on the Transformer model [9].

## 4.2 Data

We use the German and Upper Sorbian language datasets from the WMT22 website ${ }^{1)}$ for our experiments. Our model training scheme follows the original division: 60,000 instances for training, 2,000 for validation, and 2,000 for testing. Details of data are shown in Table 1.

Table 1 Statistics on WMT 2022 German-Upper Sorbian

## 5 Results

## 5.1 Analysis of different thresholds

We obtain 120,000 parallel sentence pairs when generating new sentences according to formulas. We set thresholds from 0.3 to 0.6 in steps of 0.05 for filtration and conduct seven series of experiments with the best weight of each formula. BLEU scores of models trained based on the augmented corpora generated by each formula at different thresholds for German to Upper Sorbian translation are shown in Figure 2.

Overall, the increase in the threshold value has a negative impact on the model, leading to a decrease in BLEU, but the threshold does not allow a more refined representation of the differences in the augmented corpus. Most models perform worst at a threshold of 0.6 as well as best at a threshold of 0.4. The best model uses the renormalized idf-weighted formula with a threshold of 0.4.

Comparing the best results for each formula, all models trained from formulas perform better than the unaugmented model and none of the formulas except the formula for averaged hapaxes scores perform lower than the baseline in BLEU. The renormalized idf-weighted formula, that was proposed last as a generalization (see Formula (9)), shows significant improvement in terms of BLEU and chrF2. Both the IDF-weighted formula and the renormalized IDFweighted formula show relatively superior performance in both directions.


Figure 2 Variation in BLEU scores when using different thresholds (German to Upper Sorbian)

## 5.2 Compositional augmentation policy

We construct a policy with $\mathrm{N}=6$ operations. Each operation is a separate formula with its optimal weight and threshold.

We compare the optimal nodes in both directions and compare with the non augmented model and the baseline model. Our policy shows significant improvements in all three metrics. This is shown in Table 2.

 & BLEU & chrF2 & TER \\

\cline { 2 - 6 } & Renorm. & 116 & $33.3 \pm 1.1$ & $59.0 \pm 0.8$ & $43.7 \pm 1.0$ \\
\cline { 2 - 6 } & Com. policy & 1,840 & $\mathbf{3 5 . 0} \pm \mathbf{1 . 1}$ & $\mathbf{6 4 . 9} \pm \mathbf{0 . 8}$ & $\mathbf{4 0 . 9} \pm \mathbf{0 . 9}$ \\

\cline { 2 - 7 } & Renorm. & 1,303 & $33.4 \pm 1.1$ & $61.1 \pm 0.8$ & $46.1 \pm 1.0$ \\
\cline { 2 - 7 } & Com. policy & 1,213 & $33.3 \pm 1.1$ & $\mathbf{6 3 . 7} \pm \mathbf{0 . 8}$ & $\mathbf{4 1 . 8} \pm \mathbf{0 . 9}$ \\

Table 2 Comparison of compositional policy with no augmentation model and baseline model

The results in Table 3 show details of our policy in the direction German to Upper Sorbian. We observe that finetuned nodes at depth 2 all have much lower perplexity than the first layer of the trained-only model, while no node arrives at depth 3. The main reason is that the amount of newly generated data in this low-resource setting is insuf- ficient.

 &  & Total path (s) & Perplex. &  \\

Table 3 Results of nodes in compositional policy (German to Upper Sorbian). Node with least perplexity is No. 35, generation path is formula for renormalized stopwords + idf-weighted formula.

## 6 Conclusion and limitations

Our results demonstrated significant improvements with our model: +1.7 in BLEU, +5.9 in chrF2, and -2.8 reduction in TER in the German to Upper Sorbian direction. In the Upper Sorbian to German direction, the improvements obtained are: +2.6 in chrF2 and -4.3 in TER.

As a limitation, with our policy, the exploration tree is difficult to generate at higher depths in this low-resource setting. Also, the cost of constructing a compositional augmentation tree is high, which leads to the issue of local optima when determining hyper-parameters.

## References

[1] Pengjie Wang, Liyan Wang, and Yves Lepage. Generating the middle sentence of two sentences using pre-trained models: a first step for text morphing. In Proceedings of the 27th annual meeting of the Association for Natural Language Processing, pp. 1481-1485, 2021.

[2] Koki Osawa and Yves Lepage. Data augmentation of parallel data for style transfer by generation of middle sentences (in japanese). In Proceedings of the 28th Annual Conference of the Association for Natural Language Processing, pp. 1376-1380, Hamamatsu, Japan, March 2021. Association for Natural Language Processing.

[3] Zhicheng Pan, Xinbo Zhao, and Yves Lepage. Sentence analogies for text morphing. In Proceedings of the workshop 'Analogies: from Theory to Applications (ATA@ICCBR 2022)', held in conjunction with the 30th International Conference on Case-Based Reasoning (ICCBR), Nancy, France, 2022.

[4] Matthew Eget, Xuchen Yang, and Yves Lepage. A study in the generation of multilingually aligned middle sentences. In Zygmunt Vetulani and Patrick Paroubek, editors, Proceedings of the 10th Language \& Technology Conference (LTC 2023) - Human Language Technologies as a Challenge for Computer Science and Linguistics, pp. 45-49, April 2023.

[5] Wenyi Tang and Yves Lepage. A dual reinforcement method for data augmentation using middle sentences for machine translation. In Masao Utiyama and Rui Wang (Research Track Co-Chairs), editors, Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pp. 48-58, Macau SAR, China, September 2023.

[6] Xiyuan Chen. Data augmentation for machine translation using the notion of middle sentences. Master's thesis, Graduate School of Information, Production and Systems, Waseda University, 2023.

[7] Shuhuai Ren, Jinchao Zhang, Lei Li, Xu Sun, and Jie Zhou. Text AutoAugment: Learning compositional augmentation policy for text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9029-9043, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

[8] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pp. 67-72, Vancouver, Canada, 2017. Association for Computational Linguistics.

[9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017.

## A Result analysis under different thresholds (the other direction)

Figure 3 Same as Figure 2, but for the other direction (Upper Sorbian to German)

## B Comparison of formulas under optimal parameters

We give the statistics of the results of models trained under each formula. Results are shown in Tables 4 and 5.

 &  &  & BLEU & chrF2 & TER \\

Table 4 Statistics of best results for each formula (German to Upper Sorbian). Scores in bold face are statistically different and higher than the baseline (no aug.)

## C Results for compositional augmentation policy (other direction)

Results of our compositional augmentation policy in the direction of Upper Sorbian to German are shown in Table 6.

 &  &  & BLEU & chrF2 & TER \\

Table 5 Same as Table 4, but in the other direction (Upper Sorbian to German)

 &  & Total path (s) & Perplex. &  \\

Table 6 Results of nodes in compositional policy (Upper Sorbian to German). Node with least perplexity is No. 10, generation path is formula for averaged hapaxes + renormalized LFW formula.