KoichiYasuoka
/

ltgbert-base-japanese-ud-goeswith

Token Classification

dependency-parsing

Model card Files Files and versions Community

KoichiYasuoka commited on 7 days ago

Commit

bcbee97

•

1 Parent(s): 45f6546

bug fix

Files changed (1) hide show

tokenizer.json +8 -4

tokenizer.json CHANGED Viewed

@@ -1010,8 +1010,7 @@
         "type": "Metaspace",
         "replacement": "▁",
         "add_prefix_space": false,
-        "prepend_scheme": "never",
-        "split": true
       },
       {
         "type": "Split",
@@ -1125,8 +1124,13 @@
         "type": "Metaspace",
         "replacement": "▁",
         "add_prefix_space": false,
-        "prepend_scheme": "never",
-        "split": true
       },
       {
         "type": "Replace",

         "type": "Metaspace",
         "replacement": "▁",
         "add_prefix_space": false,
+        "prepend_scheme": "never"
       },
       {
         "type": "Split",
         "type": "Metaspace",
         "replacement": "▁",
         "add_prefix_space": false,
+        "prepend_scheme": "never"
+      },
+      {
+        "type": "Strip",
+        "content": " ",
+        "start": 1,
+        "stop": 0
       },
       {
         "type": "Replace",