# SIUCER: Self Data Augmentation for Out-of-Scope Detection in Dialogues

Chunpeng Ma and Takuya Makino
Megagon Labs, Tokyo, Recruit Co., LTD.
\{ma. chunpeng, makino\}@megagon.ai
}

\begin{abstract}
Detecting out-of-scope (OOS) utterances is crucial in task-oriented dialogue systems, but obtaining enough annotated OOS dialogues to train a binary classifier directly is difficult in practice. Existing data augmentation methods generate OOS dialogues automatically, but their performance usually depends on an external corpus. Herein we propose SILVER, a self data augmentation method that does not use external data. It improves the accuracy of OOS detection (false positive rate: $90.5 \% }arrow 47.4 \%$ ). Furthermore, SILVER successfully generates high-quality in-domain (IND) OOS dialogues in terms of naturalness (percentage: $8 \% }arrow 68 \%$ ) and OOS correctness (percentage: $74 \% }arrow 88 \%$ ), as evaluated by human workers.
\end{abstract
## 1 Introduction

Task-oriented dialogue systems [2, 3] require human operators to deal with intentions that are beyond their capacities, raising the issue of out-of-scope (OOS) detection.

Due to the lack of OOS annotations in open-world settings, previous research usually detects OOS samples indirectly resorting to in-scope (INS) samples. Recently, data augmentation methods $[4,5]$ have made it possible to detect OOS directly using a binary classifier. One such method is GOLD [6]. It uses simple rules to replace utterances in known OOS dialogues with sentences selected from a large pool. However, the dependence on external corpora prevented the realization of GOLD' $s$ full potential.

We propose a method called Self Iterative OOS Labeling via Ensembling Trees (SILVER), overcoming issues of GOLD. It build pools from training data, detects OOS

This is a shortened version of the paper [1] published in the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.

Figure 1 Comparison of GOLD and SILVER. To automatically generate an OOS dialogue, SILVER replaces the first utterance with an IND utterance, while GOLD replaces the third utterance with an OOD utterance, making the dialogue become OOD and incomprehensible.

using an ensemble of decision trees [7], and generates OOS dialogues iteratively. Figure 1 compares dialogues generated by GOLD and SILVER.

## 2 GOLD: Generating Out-of-scope Labels with Data Augmentation

GOLD [6] is the data augmentation method most closely related to this work. Given a small set of annotated OOS dialogues (1\% of the size of INS), GOLD replaces utterances with sentences selected from an external pool to generate new OOS dialogues. Selected sentences should be in the neighborhood of the original utterances. Then GOLD filters the generated OOS, and combines predictions of different methods via majority voting. Filtered OOS dialogues are concatenated with the original annotated OOS dialogues and are used to train a binary classifier.

GOLD has a practical appeal. Labor-intensive data col-
lection and annotation of OOS are unnecessary, and the data augmentation method is orthogonal to the classification improvements. Both advantages extend its applicability to real scenarios. However, dependence on external corpora limits its performance.

## 3 SILVER: Methodology

Figure 2 outlines the framework of SILVER. First, we sample a small set of dialogues from the training data. These dialogues are known OOS. Then candidates are generated by randomly choosing one utterance from seed OOS dialogues and swapping it with an utterance extracted from INS (ref. §3.1). After generating numerous candidates, an ensemble classifier is used for election (ref. §3.2). Selected dialogues are concatenated to seed OOS samples, increasing the number of available OOS dialogues. Iterating this process several times provides sufficient data to train a binary classifier for OOS detection (ref. §3.3).

## 3.1 Self candidate generation

Candidates are generated by swapping utterances in seed OOS dialogues with those in the pool. To achieve this, two questions must be answered.

How should the utterance pool be built? All utterances of INS dialogues in the training data are used to build the utterance pool because we aim to generate candidates without using external corpora. Furthermore, for a task-oriented dialogue system, we assume that utterances from the user and system are in different clusters. Hence, two pools are built: (1) one for system utterances and (2) one for user utterances.

How should an appropriate utterance be selected? Two criteria are considered to determine appropriate utterances: (1) high similarity to the original utterance and (2) high divergence between each other. Figure 3 illustrates their trade-off.

(1) High similarity: By selecting utterances similar to the original one, the naturalness of the original OOS dialogue is kept. This means that the blue points in Figure 3 , which were selected by GOLD, will not be selected by SILVER.

(2) High divergence: Dialogues generated by simply modifying some words in the original utterances do not improve the classification performance. We hope the generated dialogues differ from each other. This means that selecting the black points in Figure 3 should be avoided.

Therefore, only appropriate utterances in the sense of high similarity and high divergence (i.e., green points in Figure 3) should be selected. In practice, utterances are selected from the set $\mathcal{N}(16)-\mathcal{N}(4)$, where $\mathcal{N}(k)$ is the set of $k$-nearest utterances from the original utterance.

## 3.2 Tree ensemble

SILVER classifies OOS candidates via the gradient tree boosting algorithm [8]. Feature sets consist of three parts.

(1) Probability-based feature. An intent classifier is trained as the supporting model. Then, given a dialogue $d$, the supporting model outputs the probability distribution over all intent labels: $\left[p_{1}(d), \ldots, p_{l}(d)\right]$, where $l$ is the number of possible intent labels, and $\forall i \in[1, l], 0 \leq p_{i}(d) \leq 1$. Based on this probability distribution, the probability-based feature is calculated as:

$
X_{\text {prob }}(d)=\left[\sigma^{-1}\left(p_{1}(d)\right), \ldots, \sigma^{-1}\left(p_{l}(d)\right)\right]
$

where $\sigma(\cdot)$ is the standard logistic function.

(2) Distance-based feature. Given a dialogue $d$, the distance-based feature is calculated as below:

$
\begin{aligned}
X_{\text {dist }}(d) & =\left[\operatorname{Dist}\left(h_{\mathrm{BERT}}(d), \overline{h_{\mathrm{BERT}}\left(\mathscr{D}_{1}\right)}\right)\right. \\
& \left.\ldots, \operatorname{Dist}\left(h_{\mathrm{BERT}}(d), \overline{h_{\mathrm{BERT}}\left(\mathscr{D}_{l}\right)}\right)\right]
\end{aligned}
$

where $\operatorname{Dist}(\cdot, \cdot)$ is the cosine distance between two vectors, and $h_{\mathrm{BERT}}(d)$ is the representation of the last hidden layer given input $d$. $\mathscr{D}_{i}$ is the collection of all dialogues with intent label $i$, and $\overline{h_{\mathrm{BERT}}\left(\mathscr{D}_{i}\right)}$ is their average.

(3) Ensemble-based feature. This is the average of the output probability distributions of three different runs by randomly dropping out different nodes of the baseline intent classifier, which is given as:

$
\begin{aligned}
& X_{\text {drop }}(d)= \\
& {\left[\frac{1}{3} \sum_{k=1}^{3} \sigma^{-1}\left(p_{1}^{k}(d)\right), \ldots, \frac{1}{3} \sum_{k=1}^{3} \sigma^{-1}\left(p_{l}^{k}(d)\right)\right],}
\end{aligned}
$

where $p_{i}^{k}(d)$ is the probability of intent label $i$ at the $k$-th run given dialogue $d$, after dropping out some nodes of the neural network. The dropout percentage is $10 \%$.

The final feature set is the concatenation of $X_{\text {prob }}, X_{\text {dist }}$ and $X_{\text {drop }}$. It is trained on sampled training data. Thus, no extra annotation is needed.

## 3.3 Iterative data augmentation

SILVER generates sufficient data in an iterative manner. After each iteration, newly generated dialogues are aggre-

Figure 2 Framework of SILVER.

Table 1 Results of OOS detection. Column "Pool" means whether external (Ext.) data (PersonaChat [9]) or internal (Int.) data (i.e., training data) is used to generate utterance pool. Column "Elect" gives different election methods: random selection (Rnd.), majority voting (MV), or tree ensemble (TE). Column "Iter." indicates whether dialogues are generated iteratively $(\boldsymbol{\checkmark})$ or not $(\boldsymbol{X})$. Therefore, Int. $+\mathrm{TE}+\checkmark$ means all components of SILVER are applied. Best and runner-up of different configurations are denoted by bold and underlined texts, respectively. Last line is copied from [6].

low similarity, high divergence high similarity, high divergence high similarity, low divergence

- There's no table at all? How about any time after $4 \mathrm{pm}$ O there's no table at all? how about any time after $3 \mathrm{pm}$. O Since I have so much free time, how about reserving a table for 2 at Lucca tomorrow, any time after $2 \mathrm{pm}$. that's a good show , . i haven't watched since a main character died .

Figure 3 Trade-off between similarity and divergence when selecting appropriate utterances from an utterance pool.

gated and considered known OOS dialogues. Then these are used to generate more dialogues in the next iteration. This iterative process generates high-quality dialogues with high efficiency.

High quality. The candidate list is kept small, and contains only appropriate (i.e., high similarity \& divergence) dialogues, which are rarely INS dialogues. When combined with a powerful ensemble classifier, the generated dialogues have a satisfactory quality.

High efficiency. Because INS dialogues rarely exist in the candidate list, many generated dialogues remain after election. Consequently, the number of available OOS

dialogues increases rapidly, reaching the target number in only a few iterations.

## 4 Experiments

## 4.1 Datasets and configurations

We conducted experiments on STAR [10], FLOW [11] and ROSTD [12] data.

The supporting model was a classifier finetuned on the task of intent classification, consisting of a pretrained BERT model ${ }^{1)}$, with two feed-forward layers above. The model inputs were the first 256 words of the dialogues. The model was optimized using the Adam algorithm [13].

We forced the size of seed OOS dialogues to be $1 \%$ of INS, and the target number of generated OOS dialogues was 24 times the seed size.

Experiment results were evaluated using the following metrics: (1) AUROC, area under receiver operating characteristic curve, (2) AUPR, area under precision-recall curve, and (3) FPR @ $\theta$, false positive rate with threshold $\theta$.


Table 2 Human evaluation results of 50 dialogues generated by GOLD and SILVER. Numbers are the percentages of real OOS/natural dialogues.

 &  \\

SILVER & 5,289 & 32,320 \\

Table 3 Numbers of unique utterances.

## 4.2 Results on OOS detection

Table 1 shows the key experiment results of SILVER for OOS detection. To verify the effectiveness of the three key components of SILVER corresponding to $\S 3.1$ to $\S 3.3$, we modified or removed some of these components. We can see that except for special cases (e.g., length-1 dialogues in ROSTD), combining all three components of SILVER achieves the best performance for OOS detection.

## 4.3 Evaluation of data quality

Next, we analyzed the quality of the generated data. It should be emphasized that the quality of generated dialogues is evaluated intrinsically not extrinsically. Specifically, we focus on evaluating (1) the quality of the generated dialogue itself and (2) the generated data as a whole. Herein the gain of the classification performance contributed by generated data is not considered. Although intrinsic high-quality does not necessarily contribute to extrinsic tasks directly, it is indispensable in practice.

To evaluate the quality of generated dialogue itself, we evaluate whether the generated dialogues are (1) OOS and (2) natural. The second row of Table 2 shows the human evaluation results of dialogues generated by SILVER. Compared with the first row, SILVER outperformed GOLD on both evaluation metrics.

To evaluate the quality of generated data as a whole, we compared the generated data of GOLD and SILVER with the original data, resulted in the following observations.

SILVER-generated data has a larger diversity. It is possible that one utterance was selected twice during generation. This reduces the diversity of the generated data. Table 3 shows the numbers of unique utterances

Figure 4 Visualization of the original and generated OOS dialogues.

in the generated data and the utterance pools. Although SILVER utilized a much smaller pool (built on training data), the generated data contains more unique utterances, indicating a larger diversity.

SILVER generated IND OOS data. An advantage of SILVER is its ability to generate INS OOS dialogues. We calculated the representations of the original OOS dialogues and the dialogues generated by GOLD or SILVER using vanilla RoBERTa [14]. Figure 4 shows the 2-dim tSNE visualization [15] of these representations along with the average Mahalanobis distances between clusters. The OOS dialogues generated by GOLD differed from the original ones, indicating that these dialogues are OOD. In contrast, the overlap between the original OOS and SILVERgenerated dialogues is large, implying that SILVER generates IND data.

## 5 Conclusion

We proposed SILVER to generate OOS dialogues without using external data. The components in SILVER are designed to overcome issues and realize the full potential of state-of-the-art augmentation methods. Using only training data, SILVER successfully generated high-quality IND OOS dialogues, which not only contributed to the improved performance of extrinsic tasks such as OOS detection, but are also natural enough intrinsically, indicating the potential for future applications.

## References

[1] Chunpeng Ma and Takuya Makino. Silver: Self data augmentation for out-of-scope detection in dialogues. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pp. 26-38, Nusa Dua, Bali, November 2023. Association for Computational Linguistics.

[2] Pawe 1 Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. MultiWOZ - a large-scale multidomain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.

[3] Ssu Chiu, Maolin Li, Yen-Ting Lin, and Yun-Nung Chen. SalesBot: Transitioning from chit-chat to task-oriented dialogues. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6143-6158, Dublin, Ireland, May 2022. Association for Computational Linguistics.

[4] Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. SSMBA: Self-supervised manifold based data augmentation for improving out-of-domain robustness. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 12681283, Online, November 2020. Association for Computational Linguistics.

[5] Evgeniia Razumovskaia, Ivan Vulić, and Anna Korhonen. Data augmentation and learned layer aggregation for improved multilingual language understanding in dialogue. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2017-2033, Dublin, Ireland, May 2022. Association for Computational Linguistics.

[6] Derek Chen and Zhou Yu. GOLD: Improving out-of-scope detection in dialogues using data augmentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 429-442, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

[7] Llew Mason, Jonathan Baxter, Peter Bartlett, and Marcus Frean. Boosting algorithms as gradient descent. Advances in neural information processing systems, Vol. 12, , 1999.

[8] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.

[9] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2204-2213, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[10] Johannes EM Mosig, Shikib Mehri, and Thomas Kober. Star: A schema-guided dialog dataset for transfer learning. arXiv preprint arXiv:2010.11853, 2020.

[11] Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. Task-Oriented Dialogue as Dataflow Synthesis. Transactions of the Association for Computational Linguistics, Vol. 8, pp. 556-571, 092020.

[12] Varun Gangal, Abhinav Arora, Arash Einolghozati, and Sonal Gupta. Likelihood ratios and generative classifiers for unsupervised out-of-domain detection in task oriented dialog. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, No. 05, pp. 7764-7771, Apr. 2020.

[13] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

[14] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.

[15] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using $\mathrm{t}$-sne. Journal of machine learning research, Vol. 9, No. 11, 2008.

## A Details of Datasets

Throughout this paper, we conducted experiments on three datasets: STAR [10], FLOW [11] and ROSTD [12]. Statistics of each dataset is shown in Table 4. All these datasets consist of task-oriented dialogues. Each dialogue consists of one or several utterances between human and chatbot/human. All utterances are in English.

Table 4 Numbers of INS/OOS dialogues in each dataset.

All these datasets follow the MIT license. Copyrights belong to their creators. Our use of these datasets was consistent with their intended use, i.e., for the research on dialogues of natural languages. All datasets are sufficiently anonymized to make identification of individuals impossible. We randomly sampled 100 dialogues, and asked human workers to check these dialogues. We found that these dialogues do not contain any information that names or uniquely identifies individual people, and do not contain offensive content.

## B Details of Experiments

We have reported key configurations in Section 4.1. In this section, we report more details of experiments to reproduce reported results.

All experiments were conducted on Google Cloud Platform. ${ }^{2)}$ The instance used for experiments contains one GPU (Nvidia T4).

For data augmentation, we implement the tree ensemble module using XGBoost library. ${ }^{3}$ ) Grid search is used for searching the best hyperparameters for tree ensemble. For STAR dataset, if we use all three types of features, it takes about 70 minutes for tree ensemble.

After data augmentation, we train a binary classifier to detect OOS dialogues. The classifier consists of a bert-base-uncased model (109 million parameters), and two feed-forward layers (231 thousand parameters). We resort to libraries (e.g., pytorchlightning, $\left.{ }^{4}\right)$ transformers, ${ }^{5)}$ etc.) to simplify implementation. For

2) https://console.cloud.google.com/

3) https://xgboost.ai

4) https://www.pytorchlightning.ai

5) https://github.com/huggingface/transformers
STAR dataset, it takes about 10 minutes for each epoch. We stop training after 13 epochs, and select the model with the largest AUROC on the development data as the final model.

For evaluation, we resort to scikit-learn library. ${ }^{6)}$ Specifically, we use roc_auc_score and average_precision_score functions to calculate AUROC and AUPR, respectively. Calculation of FPR @ $\theta$ is implemented by ourselves, by simply combining roc_curve function with binary search.

## C Details of Human Annotation

To evaluate the quality of automatically generated dialogues, we randomly sampled 50 dialogues and ask human workers to check them manually. Human evaluation results have been reported in Table 2. In this section, we report more details of human annotation.

For OOS correctness, we gave annotators the following instruction.

Is this dialogue really out-of-scope? For example, the chatbot can only deal with hotel reservation, but the customer asks today's weather. Another example is that the customer becomes angry because the chatbot cannot understand his/her intention.

For naturalness, we gave annotators the following instruction.

Does the replaced utterance make the whole dialogue strange? Specifically, if the dialogue remains to be natural after replacing by the new utterance, then label this dialogue as "natural," otherwise, label this dialogue as "unnatural."

All annotators are full-time employees affiliated in the same team as the authors. They all have high levels of English proficiency, and are able to annotate dialogues correctly. Annotation was done in an in-house environment, and all dialogues are used only for the purpose of research. After annotation, no ethics issues were reported.
