# How nearly random mutations to sentences affect their acceptabilities:

Preliminary quantitative analyses based on ARDJ data

Kow Kuroda ${ }^{1}$, Hikaru Yokono $^{2}$, Keiga Abe ${ }^{3}$, Tomoyuki Tsuchiya ${ }^{4}$, Yoshihiko Asao ${ }^{5}$,
Yuichiro Kobayashi ${ }^{6}$, Toshiyuki Kanamaru ${ }^{7}$, and Takumi Tagawa ${ }^{8}$
${ }^{1}$ Kyorin University, ${ }^{2}$ Meisei University, ${ }^{3}$ Wako University, ${ }^{4}$ Kyushu University,
 }^{5}$ National Institute of Communications and Information Technology, ${ }^{6}$ Nihon University, ${ }^{7}$ Kyoto University, ${ }^{8}$ Tsukuba University


\begin{abstract}
ARDJ s2u data consists of 300 sentences paired with acceptability ratings. Out of them, 36 are "originals" that were human-made and other 264 are their "mutations" that were generated either by nearly random replacement of a nominal, a verbal or a positional (e.g., case-marker) or by random swapping a pair of phrases (i.e., bunsetu) inside. This allows us to analyze effects of mutation systematically. With DBSCAN of the originals, layered analysis of mutation effects was carried out. Findings are: i) No mutations improve low acceptabilities; ii) The order of systematicity is: $\mathrm{p}<\mathrm{v}<\mathrm{n}<\mathrm{s}$; iii) The order of effect size is: $\mathrm{p}<$ $\mathrm{v}, \mathrm{n}, \mathrm{s}$, so that $\mathrm{n}-$, s-mutations often cause as large effects as v-mutation, or oven larger; iv) No simple dichotomy is possible between "acceptable" and "acceptable" sentences. Rather, their distinction is continuous and graded, so that (logistic) regression is a better modeling of the distinction.
\end{abstract
## 1 Introduction

Linguistics is concerned with acceptabilities of sentences, if not exclusively. But acceptability is an elusive notion $[1,2,3,4,5,6]$. Any group of linguists rarely agree on its theoretical definition, meaning that it is a highly theoryladen notion [7]. If its theoretical definition is somehow shared, its operative definition is virtually missing. The situation is still worse. Even if presume its definition is fine, we are still far from knowing what acceptabilities really are. No unbiased, large-scale collection of the required data has ever been done. In particular, we are far from fully understanding what modifications to a sentence bring about what effect, and to what degree, because data-driven study into this problem has never been done. What linguists know about acceptability suffers from "confirmationbias" $[8,9,10,11,12,13]$. This is the issue that dictates this research based on Acceptability Rating Data for Japanese (ARDJ for short) [14, 15, 16, 17, 18, 19].

ARDJ is a project of empirically-oriented language science, rather than mere linguistics, that attempts to implement the framework of randomized control trials in the field of linguistics, with sympathy for Evidence-based Medicine
(EBM) [20,21]. A crucial conviction that paved the way for ARDJ is that intuitions by experts (e.g., acceptability judgments by linguists) are valuable with no doubt, but they are too weak "evidence" to be treated as "facts"; rather, they are more or less reliable "guides."

The stimuli used in the ARDJ surveys comprise two kinds of sentences: 36 "originals" hand-crafted by humans and their "variants" generated by applying nearly random mutations on the originals. Mutations are only nearly random rather than fully random, because they have target sites, and values for replacement are chosen based on word2vec distributional similarities. See [16] for relevant details. Appendix A. 1 gives a brief summary. Effects of mutation can be systematically analyzed by comparing responses to the originals and their mutations.

## 2 Analysis and results

To make our analysis effective, we use PCA, tSNE with DBSCAN clustering, overplot, and KL-divergence-based heatmap/clustering. In $\$ 2.3 .1-\$ 2.3 .4$, analysis of effects of mutation is implemented as a layered analysis in which Clusters 1, 2, 3 and 4, are taken as four separate groups of data. Discarding the originals in Clusters 2, 3 and 4, KLdivergence-based analysis is performed on mutations to the originals in Cluster 1 in \$2.4. Lastly, effects are analyzed by exploring properties of mutation-wise KL-divergence heatmaps in $\S 2.5$.

## 2.136 originals and their attributes

ARDJ s2u data [17] has 36 "originals" presented in Figure $1,001,002, \ldots$, o36, with relevant attributes. Details are presented in §2.1. How this data was prepared is described in Appendix A. The originals are sorted decreasingly by $\mathrm{r} 01, \mathrm{r} 12$ values, and increasingly by $\mathrm{r} 23, \mathrm{r} 3 \mathrm{x}$ values. Only 32 originals have more than 2 mutations. So, the number of effective originals is 32 rather than 36 , as Figure 1 shows. The effective ones have $(8.31-1)$ mutations/variants on average, with a maximum of 16 and minimum of 4 .

Figure 1: $\mathbf{3 6}$ originals with attributes

The originals were clustered using DBCAN, giving us 4 clusters. $\$ 2.2$ deals with this result. The resulting clustering is laid out over a PC1-PC2 plane of PCA rotation and a Dim1-Dim2 plane of tSNE transformation. This gives us how the sentences are associated to each other.

## 2.2 Clustering of originals and all stimuli

Figure 2: PCA and tSNE of 36 originals

## 2.2.1 Clustering originals only

DBSCAN was used to cluster the 36 originals to identify 4 clusters, which are shown in PCA and tSNE plots in Figure 2. tSNE in Figure 2 suggests the existence of a unified dimension along which acceptability can be measured. The deviance of Cluster 2 from it can be accounted for its special nature described below.

## 2.2.2 Clustering all 270 stimuli

For comparison, we present clustering of all stimuli including the 36 originals described above. Figure 3 gives PCA and tSNE plots of DBSCAN clustering. Note that the setting for eps and minPts parameters here is independent from those for the parameters for the DBSCAN clustering of the 36 originals.

Figure 3: PCA and tSNE of all 270 stimuli

## 2.3 Layered Analysis

This clustering result seems natural in that clusterwise overplots in Figure 4 seem to capture both the similarities and dissimilarities quite well. Each cluster consists of instances with surprising similarities.

rating ranges
lustered by DBSCAN (eps $=$

rating ranges
DBSCAN (eps $=0$

Figure 4: Clusterwise ovwerplots of originals

Comparing with the data in Figure 1, we can safely conclude the following:

a. Cluster 1 identifies the group of "acceptable" stimuli, Cluster 4 the one of "unacceptable" stimuli, and Clusters 2 and 3 the ones of "ambiguous" stimuli, with different qualities.

b. Cluster 3 consists of 3 instances, 003, 011 and o24, that have a "split" decision, in that raters are split into two groups of those who accept them fully and those who reject them fully.

c. Cluster 4 consists of 3 instances, 001,007 and 032, that have a nearly "uniform" distribution of (un)acceptability along the range, though with a slight inclination toward acceptability.

With these distinctions among the originals, we can now turn to effectively analyzing the effects of mutation to answer the research question: how random mutations to sentences affect their acceptabilities, and to what extent.

## 2.3.1 Analysis 1a: Mutations to originals in Cluster 4

We start our analysis with the one of the mutations to the originals to Cluster 4, a relatively large cluster, having 13
members which can be labelled as "unacceptable" stimuli. This is because the effects are simple and can be characterized most easily.

Effects of mutation are analyzed in three visualizations, as shown in Figure 5: i) overplot of response profiles (in the leftmost column); ii) proportional distribution of range values after reordering; and iii) layout of DBSCAN clusters in a rotated geometry $G$.

Figure 5: Mutations to originals to Cluster 4 [4 samples]

In each analysis, stimuli are ordered by their range values. So, their positions relative to o in the legend roughly indicate if mutations increase or decrease acceptabilities.

As suggested by the plots in Figure 5, virtually no mutations improve the original acceptabilities. In addition, the response profiles, i.e., shapes of distribution, stay unchanged for most cases. So, there seems to be a rule like "once (made) deviant, always deviant." This finding is not truly surprising, let alone unexpected; yet, it deserves a mention because this is the first empirical confirmation of a truism on a large scale that has long awaited testing.

## 2.3.2 Analysis 1b: Mutations to originals in Cluster 3

Effects of mutations to cases in Cluster 3 are similar to the ones to stimuli in Cluster 4. Stimuli in Cluster 3 almost are almost never improved by any mutation, though, admittedly, we can hardly state generalizations safely because we have too few affected cases.

## 2.3.3 Analysis 1c: Mutations to originals in Cluster 2

Effects of mutations to cases in Cluster 2 are somewhat different from the ones to stimuli in Cluster 3. Unlike stimuli in Cluster 3, stimuli in Cluster 2 sometimes improve, though, again, we can hardly state generalizations safely

Figure 6: Mutations to originals to Cluster 3 [One case was dropped due to insufficient cases.]

due to having too few affected cases. It is interesting that o11 and o24 get more acceptable through certain mutations, though o03 does not possess such properties.

Figure 7: Mutations to originals to Cluster 2 [2 samples]

## 2.3.4 Analysis 1d: Mutations to originals in Cluster 1

Mutations to the originals in Clusters 2, 3 and 4 tell us virtually nothing interesting about how mutations affect acceptability, because we can hardly identify meaningful differences between mutations and their originals. This is not true of mutations to the originals in Cluster 1.

Mutation effects are different case by case, but generally speaking, we can observe the following:

a. Lexical changes (i.e., mutations to either $\mathrm{n}$ (ominals), v(erbals), or $\mathrm{p}$ (ositionals)) nearly always drop acceptability, while non-lexical changes (i.e., swapping) do not. Swapping often keeps the acceptability.

b. It is not true, at least not observed, that n-changes reduce acceptability more than $\mathrm{v}$-and $\mathrm{p}$-changes, as far as degree of deviance is measured.

c. Changes to $\mathrm{p}$ introduce only a slight amount of deviance, though it is unignorable.

d. Effects of mutation are diverse, in that the number of clusters recognized is greater than the ones in other clusters, Clusters 2, 3 and 4.

e. Mutations do not simply make sentences unacceptable, in that resulting profiles do not always look like those in the originals in Cluster

Figure 8: Mutations to originals to Cluster 1 [5 samples]

4. Rather, they look more like the originals in Cluster 3 or 2 . So, the effects of mutation are more subtle than researchers have guessed.

With their relative importance, cases in Cluster 1 deserve further exploration. We do it by analyzing them with KLdivergence cross-table.

## 2.4 Analysis 2: KL-divergence for originals to Cluster 1

The diagrams in Figure 9 are heatmaps of the 9 samples, taken in Figure 8, allowing us to make the following observations:

a. p- and v-mutations tend to cluster together, but not always.

b. swapping and n-mutations tend to lack systematic effects, though they often cause drastic deterioration.

## 2.5 Analysis 3: KL-divergence cross-table by mutation

Different types of mutation, i.e., p, v, $n$ and $s$, have different magnitudes of effects. To estimate them, we can use KLdivergence heatmaps, as shown in Figure 10.

If the contrasts between "hot" and "cool" areas are sharp, the magnitude of effects can be judged to be large. If the

Figure 9: KL-div. heatmaps for Cluster 1 cases [009, o08, $005,016,023,022,026,027,031]$

Figure 10: KL-div. heatmaps for mutation types [p, v, n, $\mathbf{s ,} \mathbf{0}]$

degree contrasts are spread, effects are unsystematic. With this criterion, we can conclude that i) the effect of $p$ mutations is least contrastive and most systematic; ii) that of $\mathrm{v}$ mutations is second least contrastive and rather systematic; iii) that of n mutations is second most contrastive and rather unsystematic; and iv) that of s mutations is most contrastive and most systematic. Note that the 36 originals are contrastive only because they are largely designed so.

## 3 Discussion and Conclusion

So, what are causes of reduced acceptabilities? With the results so far, we can now decide if we are able to answer this question. The most sincere answer to this would be: there is no simple answer to this question. What does this mean, then?

Simple dichotomy between "acceptable" and "unacceptable" kinds of sentences is too simple and unrealistic, though it is widely accepted and practiced in many areas of linguistics. What really matters about acceptabilities are response profiles (distributional pattern of responses) rather than categorical judgments, on the one hand, and regression to response profiles rather than binary classification (e.g., dichotomy $=$ categorical judgment), on the other. This is a conclusion compatible with the results in [15].

What do linguists need to do with this reality, then? We would suggest that the goals of linguistics should be redefined. Redefined goals include the construction of a mapping from sentences in textual form (e.g., values in s.text in Table 1) to response profiles (e.g., arrays of values [r01, r12, r23, r3x] in the table in Figure 1), yielding overplots in Figures 4. This can never be attained until large-scaled, bias-free databases like ARDJ are built for as many languages as possible. Until then, linguistics would never make itself an empirical science.

## Acknowledgements

This research was supported by JSPS through Grant-in-Aid (16K13223)

We are grateful for Shunji Awazu (Jissen Women's University), Asuka Terai (Future University of Hakodate) and Minoru Yamaizumi (Osaka University), who kindly helped our execution of Phase 1 of ARDJ Survey 2.

All statistic analyses were done on $\mathrm{R}$ version 3.6.3, a free statistical software developed by R Core Team .

## References

[1] Wayne Cowart. Experimental Syntax: Applying Objective Methods to Sentence Judgments. Sage Publishing, 1997.

[2] Ira A. Noveck and Dan Sperber, editors. Experimental Pragmatics. Palgrave Macmillan, 2005.

[3] Gary Dean Prideaux, Bruce L. Derwing, and William J. Baker. Experimental Linguistics: Integration of Theories and Applications. John Benjamins, 1979.

[4] Carson T. Schütze. The Empirical Base of Linguistics: Grammaticallity Judgments and Linguistic Methodlogy. Language Science Press/University of Chicago Press, 2016 [1996].

[5] Jon Sprouse. Three open questions in experimental syntax. Linguistics Vanguard, Vol. 1, No. 1, pp. 89-100, 2015.

[6] Jon Sprouse and Norbert Horstein. Experimental Syntax and Island Effects. Cambridge University Press, 2013.

[7] Holly P. Branigan and Martin Pickering. An experimental approach to linguistic representation. Behavioral and Brain Sciences, Vol. 40, 2017.

[8] Jonathan Baron. Thinking and Deciding. Cambridge University Press, 2000 [1988, 1994].

[9] H. W. Bierhoff and R. Klein. Expectations, confirmation bias, and suggestibility. In V. A. Gheorghiu, P. Netter, H. J. Eysenck, and R. Rosenthal, editors, Suggestion and Suggestibility, pp. 337-346. New York: Springer, 1989.

[10] Joshua Klayman. Varieties of confirmation bias. Psychology of Learning and Motivation, Vol. 32, pp. 384-418, 1995.

[11] Thomas Gilovich. How We Know What Isn't So: The Fallibility of Human Reason in Everyday Life. Free Press., 1993. [(日本語訳) 人間，この信じやすきもの: 迷信・誤信はど うして生まれるか. 新曜社.].

[12] David Robson. The Intelligence Trap: Why Smart People Do Stupid Things and How to Make Wiser Decisions. Hodder \& Stoughton, 2019.

[13] Stuart A. Vyse. Believing in Magic: The Psychology of Superstition. Oxford University Press, reprinted edition, 2000. [(日本語訳): 人はなぜ迷信を信じるのか: 思い込みの心理学. 朝日新聞社, 1999.].

[14] Kow Kuroda. Reassessing PCA of Acceptability Rating Data for Japanese (ARDJ) using kernel MVA. In Proceedings of the 26th Annual Meeting of Association of NLP, pp. 15231526,2020
[15] Kow Kuroda, Hikaru Yokono, Keiga Abe, Tomoyuki Tsuchiya, Yoshihiko Aaso, Yuichiro Kobayashi, Toshiyuki Kanamaru, and Takumi Tagawa. Simulating acceptability judgments using ARDJ data. In Proceedings of the 27th Annual Meeting of Association of NLP, pp. 739-743, 2021.

[16] Kow Kuroda, Hikaru Yokono, Keiga Abe, Tomoyuki Tsuchiya, Yoshihiko Asao, Yuichiro Kobayashi, Toshiyuki Kanamaru, and Takumi Tagawa. Development of Acceptability Rating Data of Japanese (ARDJ): An initial report. In Proceedings of the 24th Annual Meeting of the Association for NLP, pp. 65-68, 2018.

[17] Kow Kuroda, Hikaru Yokono, Keiga Abe, Tomoyuki Tsuchiya, Yoshihiko Asao, Yuichiro Kobayashi, Toshiyuki Kanamaru, and Takumi Tagawa. Insights from a large scale web survey for Acceptability Rating Data for Japanese (ARDJ) project. In Proceedings of the 25th Annual Meeting for the Association of NLP, pp. 253-256, 2019.

[18] 黒田航, 阿部慶賀, 横野光, 土屋智行, 小林雄一郎, 金丸敏幸, 浅尾仁彦, 田川拓海. 容認度評定に影響する要因の定量的評価: 日本語容認度評定データ (ARDJ) から得られた知見. 日本認知科学会第 36 回大会発表論文集, pp. 727-736, 2019.

[19] 黒田航, 阿部慶賀, 粟津俊二, 寺井あすか, 土屋智行. ARDJ を基にした容認度評定値と反応時間の多変量解析と混合回帰分析. 認知科学会第 37 回大会発表論文集, pp. 919-928, 2020.

[20] G. Evidence-based Medicine Group [Guyatt, J. Cairns, and D. et al.] Churchill. Evidence-based medicine: A new approach to teaching the practice of medicine. Journal of the American Medical Association, Vol. 268, No. 17, pp. 2420 5, 1992 (Nov 4).

[21] David Isaacs and Dominic Fitzgerald. Seven alternatives to evidence based medicine. The British Medical Journal, Vol. 319, No. 7225, p. 1618, 1999.

## A Data preparation

## A. 1 Construction of stimuli

To make the outcomes of ARDJ truly empirical and useful, we need to employ as unbiased stimuli as possible. The best way to implement it is to employ automatic generation rather than relying too much on human intuitions. Random mutation in DNA sequences serves as a model. The basic idea is the following:

(4) Steps of randomized generation with targets

Step 1. Construct sentences, called "originals," $O=\left.\{o_{1}\right.$, $\left.o_{2}, \ldots, o_{n}\right.\}$ with or without deviance.

Step 2. Introduce random mutations to instances of $O$. Let $M$ denote the result.

Step 3. Mix $O$ and $M$ and use its subsets for acceptability rating tasks.

More details of Step 2 are given in the following:

(5) A) Random replacement of a lexical item under POSidentity (edit type: 1(exical)); B) Random replacement of a postpositional case-marker $(\mathrm{P})$ with another (edit type: $\mathrm{p}($ ositional)); C) Random positional exchange of a given pair of NPs (or PPs) (edit type: s(wapping)).

provided that items for replacement are selected on a predefined measure such as distributional similarity.

The first and second authors developed Python scripts to implement mutations of $\mathrm{A}-, \mathrm{B}-$ and $\mathrm{C}$-classes. They are provided at https://github.com/kow-k/ Japanese-sentence-mutators.

## A. 2 Surveys

By 2019, ARDJ completed two experiments. The first one, called "Survey 1," was carried out in 2017. It was intended to be a pilot study with only a limited variety of responders (roughly 200 college students only) on 200 sentences for stimuli. Kuroda et al. [16] reported on this survey. The second experiment, called "Survey 2," was carried out in 2018. This was the main study, with the stimulus set expanded to 300 with some overlap with Survey 1.

Survey 2 had two phases, Phases 1 and 2, and collected responses from 1,880 participants in total. Phase 1 was a small scale, paper-based survey, to which 201 participants (mostly all college or university students) contributed responses. This was comparable to the pilot study done in 2017. Phase 2 was a large scale web survey to which 1,679 participants contributed responses. They were significantly more varied in their attributes and we would safely state responses obtained were randomized enough. Survey 2 unified (s2u) data consists of unification of responses from Phase 1 and Phase 2. These data are freely available at https://kow-k.github.io/ Acceptability-Rating-Data-of-Japanese/, but you need to register to use them.

## A. 3 Collection of ratings/responses

On acceptability rating, participants were asked to choose one of the four choices in (6).

(6) 0. 違和感がなく自然に理解できる文 [natural and easy to understand]; 1. 違和感を感じるが理解可能な文 [more or less deviant but comprehensible]; 2. 違和感を感じ て理解困難な文 [deviant and difficult to understand]; 3.不自然な理解不能な文 [quite unnatural and incomprehensible]

Prefixes 0, 1, 2 and 3 are added to indicate the degrees of deviance, though they need not be on a single scale.

Outlier responders were filtered out using standard deviation $(0.6<\mathrm{sd}<1.5)$ and Mahalanobis distance $(<0.95)$. See Kuroda et al [17] for relevant details.

## A. 4 Standardizing responses

Note that gr0, .., gr9 are different data sets, and cannot be directly compared. Comparison of them requires standardization. All groups were collapsed and responses were counted for each of the four rating ranges $r[0,1), r[1,2), r[2,3)$, and $r[3, \infty)$.

Table 1 shows 10 samples of this process. These raw counts were then converted into proportions to item-wise sums. This gave us density array, $P=$ $\langle p[0,1), p[1,2), p[2,3), p[3, \infty)\rangle$, as exemplified in Table 2. Note that allowed response values were $0,1,2$, and 3 . These numbers are reinterpreted as ranges $r[0,1), r[1,2)$, $r[2,3)$, and $r[3, \infty)$, respectively, where $r[i, j)$ means a sum of response counts between $i$ and $j$ with $i$ included and $j$ excluded. The converted arrays of ranged response probabilities in this format are to be called "response potentials." They are commensurable among groups with different sets of responders, and were used as effective encodings of the responses in a variety of analyses.

Table 1: Frequency table by ranges (6 samples)

Table 2: Density table by ranges (6 samples)
 Preliminary quantitative analyses based on ARDJ data  Kow Kuroda ${ }^{1}$, Hikaru Yokono $^{2}$, Keiga Abe ${ }^{3}$, Tomoyuki Tsuchiya ${ }^{4}$, Yoshihiko Asao ${ }^{5}$, Yuichiro Kobayashi ${ }^{6}$, Toshiyuki Kanamaru ${ }^{7}$, and Takumi Tagawa ${ }^{8}$ ${ }^{1}$ Kyorin University, ${ }^{2}$ Meisei University, ${ }^{3}$ Wako University, ${ }^{4}$ Kyushu University,  }^{5}$ National Institute of Communications and Information Technology, ${ }^{6}$ Nihon University, ${ }^{7}$ Kyoto University, ${ }^{8}$ Tsukuba University   \begin{abstract} ARDJ s2u data consists of 300 sentences paired with acceptability ratings. Out of them, 36 are "originals" that were human-made and other 264 are their "mutations" that were generated either by nearly random replacement of a nominal, a verbal or a positional (e.g., case-marker) or by random swapping a pair of phrases (i.e., bunsetu) inside. This allows us to analyze effects of mutation systematically. With DBSCAN of the originals, layered analysis of mutation effects was carried out. Findings are: i) No mutations improve low acceptabilities; ii) The order of systematicity is: $\mathrm{p}<\mathrm{v}<\mathrm{n}<\mathrm{s}$; iii) The order of effect size is: $\mathrm{p}<$ $\mathrm{v}, \mathrm{n}, \mathrm{s}$, so that $\mathrm{n}-$, s-mutations often cause as large effects as v-mutation, or oven larger; iv) No simple dichotomy is possible between "acceptable" and "acceptable" sentences. Rather, their distinction is continuous and graded, so that (logistic) regression is a better modeling of the distinction. \end{abstract 