Hugging Face's logo

alistairmcleay
/
UBAR-distilgpt2

Text Generation
Transformers
PyTorch
gpt2
text-generation-inference
Inference Endpoints
Model card Files Community
1
alistairmcleay commited on
Commit
6c2afd3
•
1 Parent(s): 1a092d4

Upload eval_log.json

Browse files
Files changed (1) hide show
  1. eval_log.json +635 -0
eval_log.json ADDED
@@ -0,0 +1,635 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
2
+ INFO:root:train size:8434, dev size:999, test size:1000
3
+ INFO:root:Generate setting:
4
+ use true_prev_bspn=False
5
+ use true_prev_aspn=False
6
+ use true_db_pointer=False
7
+ use true_prev_resp=False
8
+ use true_curr_bspn=False
9
+ use true_curr_aspn=False
10
+ use_all_previous_context=True
11
+ INFO:root:***** Running Evaluation *****
12
+ INFO:root: Num Turns = 7372
13
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
14
+ INFO:root:train size:8434, dev size:999, test size:1000
15
+ INFO:root:Generate setting:
16
+ use true_prev_bspn=False
17
+ use true_prev_aspn=False
18
+ use true_db_pointer=False
19
+ use true_prev_resp=False
20
+ use true_curr_bspn=False
21
+ use true_curr_aspn=False
22
+ use_all_previous_context=True
23
+ INFO:root:***** Running Evaluation *****
24
+ INFO:root: Num Turns = 7372
25
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
26
+ INFO:root:train size:8434, dev size:999, test size:1000
27
+ INFO:root:Generate setting:
28
+ use true_prev_bspn=False
29
+ use true_prev_aspn=False
30
+ use true_db_pointer=False
31
+ use true_prev_resp=False
32
+ use true_curr_bspn=False
33
+ use true_curr_aspn=False
34
+ use_all_previous_context=True
35
+ INFO:root:***** Running Evaluation *****
36
+ INFO:root: Num Turns = 7372
37
+ INFO:root:inference time: 2.18 min
38
+ INFO:root:Scoring time: 0.02 min
39
+ INFO:root:validation [CTR] match: 93.00 success: 81.00 bleu: 14.31 score: 101.31
40
+ INFO:root:update eval results to logs_test/home-all_test.json
41
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
42
+ INFO:root:train size:8434, dev size:999, test size:1000
43
+ INFO:root:Generate setting:
44
+ use true_prev_bspn=False
45
+ use true_prev_aspn=False
46
+ use true_db_pointer=False
47
+ use true_prev_resp=False
48
+ use true_curr_bspn=False
49
+ use true_curr_aspn=False
50
+ use_all_previous_context=True
51
+ INFO:root:***** Running Evaluation *****
52
+ INFO:root: Num Turns = 7372
53
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
54
+ INFO:root:train size:8434, dev size:999, test size:1000
55
+ INFO:root:Generate setting:
56
+ use true_prev_bspn=False
57
+ use true_prev_aspn=False
58
+ use true_db_pointer=False
59
+ use true_prev_resp=False
60
+ use true_curr_bspn=False
61
+ use true_curr_aspn=False
62
+ use_all_previous_context=True
63
+ INFO:root:***** Running Evaluation *****
64
+ INFO:root: Num Turns = 7372
65
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
66
+ INFO:root:train size:8434, dev size:999, test size:1000
67
+ INFO:root:Generate setting:
68
+ use true_prev_bspn=False
69
+ use true_prev_aspn=False
70
+ use true_db_pointer=False
71
+ use true_prev_resp=False
72
+ use true_curr_bspn=False
73
+ use true_curr_aspn=False
74
+ use_all_previous_context=True
75
+ INFO:root:***** Running Evaluation *****
76
+ INFO:root: Num Turns = 7372
77
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
78
+ INFO:root:train size:8434, dev size:999, test size:1000
79
+ INFO:root:Generate setting:
80
+ use true_prev_bspn=False
81
+ use true_prev_aspn=False
82
+ use true_db_pointer=False
83
+ use true_prev_resp=False
84
+ use true_curr_bspn=False
85
+ use true_curr_aspn=False
86
+ use_all_previous_context=True
87
+ INFO:root:***** Running Evaluation *****
88
+ INFO:root: Num Turns = 7372
89
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
90
+ INFO:root:train size:8434, dev size:999, test size:1000
91
+ INFO:root:Generate setting:
92
+ use true_prev_bspn=False
93
+ use true_prev_aspn=False
94
+ use true_db_pointer=False
95
+ use true_prev_resp=False
96
+ use true_curr_bspn=False
97
+ use true_curr_aspn=False
98
+ use_all_previous_context=True
99
+ INFO:root:***** Running Evaluation *****
100
+ INFO:root: Num Turns = 7372
101
+ INFO:root:inference time: 0.62 min
102
+ INFO:root:Scoring time: 0.00 min
103
+ INFO:root:validation [CTR] match: 100.00 success: 100.00 bleu: 18.49 score: 118.49
104
+ INFO:root:update eval results to logs_test/experiments-all_test.json
105
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
106
+ INFO:root:train size:8434, dev size:999, test size:1000
107
+ INFO:root:Generate setting:
108
+ use true_prev_bspn=False
109
+ use true_prev_aspn=False
110
+ use true_db_pointer=False
111
+ use true_prev_resp=False
112
+ use true_curr_bspn=False
113
+ use true_curr_aspn=False
114
+ use_all_previous_context=True
115
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
116
+ INFO:root:train size:8434, dev size:999, test size:1000
117
+ INFO:root:Generate setting:
118
+ use true_prev_bspn=False
119
+ use true_prev_aspn=False
120
+ use true_db_pointer=False
121
+ use true_prev_resp=False
122
+ use true_curr_bspn=False
123
+ use true_curr_aspn=False
124
+ use_all_previous_context=True
125
+ INFO:root:***** Running Evaluation *****
126
+ INFO:root: Num Turns = 7372
127
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
128
+ INFO:root:train size:8434, dev size:999, test size:1000
129
+ INFO:root:Generate setting:
130
+ use true_prev_bspn=False
131
+ use true_prev_aspn=False
132
+ use true_db_pointer=False
133
+ use true_prev_resp=False
134
+ use true_curr_bspn=False
135
+ use true_curr_aspn=False
136
+ use_all_previous_context=True
137
+ INFO:root:***** Running Evaluation *****
138
+ INFO:root: Num Turns = 7372
139
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
140
+ INFO:root:train size:8434, dev size:999, test size:1000
141
+ INFO:root:Generate setting:
142
+ use true_prev_bspn=False
143
+ use true_prev_aspn=False
144
+ use true_db_pointer=False
145
+ use true_prev_resp=False
146
+ use true_curr_bspn=False
147
+ use true_curr_aspn=False
148
+ use_all_previous_context=True
149
+ INFO:root:***** Running Evaluation *****
150
+ INFO:root: Num Turns = 7372
151
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
152
+ INFO:root:train size:8434, dev size:999, test size:1000
153
+ INFO:root:Generate setting:
154
+ use true_prev_bspn=False
155
+ use true_prev_aspn=False
156
+ use true_db_pointer=False
157
+ use true_prev_resp=False
158
+ use true_curr_bspn=False
159
+ use true_curr_aspn=False
160
+ use_all_previous_context=True
161
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
162
+ INFO:root:train size:8434, dev size:999, test size:1000
163
+ INFO:root:Generate setting:
164
+ use true_prev_bspn=False
165
+ use true_prev_aspn=False
166
+ use true_db_pointer=False
167
+ use true_prev_resp=False
168
+ use true_curr_bspn=False
169
+ use true_curr_aspn=False
170
+ use_all_previous_context=True
171
+ INFO:root:***** Running Evaluation *****
172
+ INFO:root: Num Turns = 7372
173
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
174
+ INFO:root:train size:8434, dev size:999, test size:1000
175
+ INFO:root:Generate setting:
176
+ use true_prev_bspn=False
177
+ use true_prev_aspn=False
178
+ use true_db_pointer=False
179
+ use true_prev_resp=False
180
+ use true_curr_bspn=False
181
+ use true_curr_aspn=False
182
+ use_all_previous_context=True
183
+ INFO:root:***** Running Evaluation *****
184
+ INFO:root: Num Turns = 7372
185
+ INFO:root:inference time: 0.70 min
186
+ INFO:root:Scoring time: 0.01 min
187
+ INFO:root:validation [CTR] match: 100.00 success: 100.00 bleu: 18.49 score: 118.49
188
+ INFO:root:update eval results to logs_test/experiments-all_test.json
189
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
190
+ INFO:root:train size:8434, dev size:999, test size:1000
191
+ INFO:root:Generate setting:
192
+ use true_prev_bspn=False
193
+ use true_prev_aspn=False
194
+ use true_db_pointer=False
195
+ use true_prev_resp=False
196
+ use true_curr_bspn=False
197
+ use true_curr_aspn=False
198
+ use_all_previous_context=True
199
+ INFO:root:***** Running Evaluation *****
200
+ INFO:root: Num Turns = 7372
201
+ INFO:root:inference time: 4.61 min
202
+ INFO:root:Scoring time: 0.01 min
203
+ INFO:root:validation [CTR] match: 100.00 success: 90.00 bleu: 12.60 score: 107.60
204
+ INFO:root:update eval results to logs_test/experiments-all_test.json
205
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
206
+ INFO:root:train size:8434, dev size:999, test size:1000
207
+ INFO:root:Generate setting:
208
+ use true_prev_bspn=False
209
+ use true_prev_aspn=False
210
+ use true_db_pointer=False
211
+ use true_prev_resp=False
212
+ use true_curr_bspn=False
213
+ use true_curr_aspn=False
214
+ use_all_previous_context=True
215
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
216
+ INFO:root:train size:8434, dev size:999, test size:1000
217
+ INFO:root:Generate setting:
218
+ use true_prev_bspn=False
219
+ use true_prev_aspn=False
220
+ use true_db_pointer=False
221
+ use true_prev_resp=False
222
+ use true_curr_bspn=False
223
+ use true_curr_aspn=False
224
+ use_all_previous_context=True
225
+ INFO:root:Running eval on test
226
+ INFO:root:***** Running Evaluation *****
227
+ INFO:root: Num Turns = 7372
228
+ INFO:root:inference time: 21.91 min
229
+ INFO:root:Scoring time: 0.17 min
230
+ INFO:root:validation [CTR] match: 87.90 success: 76.60 bleu: 15.30 score: 97.55
231
+ INFO:root:update eval results to logs_test/home-all_test.json
232
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
233
+ INFO:root:train size:8434, dev size:999, test size:1000
234
+ INFO:root:Generate setting:
235
+ use true_prev_bspn=False
236
+ use true_prev_aspn=False
237
+ use true_db_pointer=False
238
+ use true_prev_resp=False
239
+ use true_curr_bspn=False
240
+ use true_curr_aspn=False
241
+ use_all_previous_context=True
242
+ INFO:root:Running eval on test
243
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
244
+ INFO:root:train size:8434, dev size:999, test size:1000
245
+ INFO:root:Generate setting:
246
+ use true_prev_bspn=False
247
+ use true_prev_aspn=False
248
+ use true_db_pointer=False
249
+ use true_prev_resp=False
250
+ use true_curr_bspn=False
251
+ use true_curr_aspn=False
252
+ use_all_previous_context=True
253
+ INFO:root:Running eval on test
254
+ INFO:root:***** Running Evaluation *****
255
+ INFO:root: Num Turns = 7372
256
+ INFO:root:Reading encoded data from ./data/preprocessed/multi-woz-processed/new_db_se_blank_encoded.data.json
257
+ INFO:root:train size:8434, dev size:999, test size:1000
258
+ INFO:root:Generate setting:
259
+ use true_prev_bspn=False
260
+ use true_prev_aspn=False
261
+ use true_db_pointer=False
262
+ use true_prev_resp=False
263
+ use true_curr_bspn=False
264
+ use true_curr_aspn=False
265
+ use_all_previous_context=True
266
+ INFO:root:Running eval on test
267
+ INFO:root:***** Running Evaluation *****
268
+ INFO:root: Num Turns = 7372
269
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
270
+ INFO:root:train size:8434, dev size:999, test size:1000
271
+ INFO:root:Generate setting:
272
+ use true_prev_bspn=False
273
+ use true_prev_aspn=False
274
+ use true_db_pointer=False
275
+ use true_prev_resp=False
276
+ use true_curr_bspn=False
277
+ use true_curr_aspn=False
278
+ use_all_previous_context=True
279
+ INFO:root:Running eval on test
280
+ INFO:root:***** Running Evaluation *****
281
+ INFO:root: Num Turns = 7372
282
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
283
+ INFO:root:train size:8434, dev size:999, test size:1000
284
+ INFO:root:Generate setting:
285
+ use true_prev_bspn=False
286
+ use true_prev_aspn=False
287
+ use true_db_pointer=False
288
+ use true_prev_resp=False
289
+ use true_curr_bspn=False
290
+ use true_curr_aspn=False
291
+ use_all_previous_context=True
292
+ INFO:root:Running eval on test
293
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
294
+ INFO:root:train size:8434, dev size:999, test size:1000
295
+ INFO:root:Generate setting:
296
+ use true_prev_bspn=False
297
+ use true_prev_aspn=False
298
+ use true_db_pointer=False
299
+ use true_prev_resp=False
300
+ use true_curr_bspn=False
301
+ use true_curr_aspn=False
302
+ use_all_previous_context=True
303
+ INFO:root:Running eval on test
304
+ INFO:root:***** Running Evaluation *****
305
+ INFO:root: Num Turns = 7372
306
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
307
+ INFO:root:train size:8434, dev size:999, test size:1000
308
+ INFO:root:Generate setting:
309
+ use true_prev_bspn=False
310
+ use true_prev_aspn=False
311
+ use true_db_pointer=False
312
+ use true_prev_resp=False
313
+ use true_curr_bspn=False
314
+ use true_curr_aspn=False
315
+ use_all_previous_context=True
316
+ INFO:root:Running eval on test
317
+ INFO:root:***** Running Evaluation *****
318
+ INFO:root: Num Turns = 7372
319
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
320
+ INFO:root:train size:8434, dev size:999, test size:1000
321
+ INFO:root:Generate setting:
322
+ use true_prev_bspn=False
323
+ use true_prev_aspn=False
324
+ use true_db_pointer=False
325
+ use true_prev_resp=False
326
+ use true_curr_bspn=False
327
+ use true_curr_aspn=False
328
+ use_all_previous_context=True
329
+ INFO:root:Running eval on test
330
+ INFO:root:***** Running Evaluation *****
331
+ INFO:root: Num Turns = 7372
332
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
333
+ INFO:root:train size:8434, dev size:999, test size:1000
334
+ INFO:root:Generate setting:
335
+ use true_prev_bspn=False
336
+ use true_prev_aspn=False
337
+ use true_db_pointer=False
338
+ use true_prev_resp=False
339
+ use true_curr_bspn=False
340
+ use true_curr_aspn=False
341
+ use_all_previous_context=True
342
+ INFO:root:Running eval on test
343
+ INFO:root:***** Running Evaluation *****
344
+ INFO:root: Num Turns = 7372
345
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
346
+ INFO:root:train size:8434, dev size:999, test size:1000
347
+ INFO:root:Generate setting:
348
+ use true_prev_bspn=False
349
+ use true_prev_aspn=False
350
+ use true_db_pointer=False
351
+ use true_prev_resp=False
352
+ use true_curr_bspn=False
353
+ use true_curr_aspn=False
354
+ use_all_previous_context=True
355
+ INFO:root:Running eval on test
356
+ INFO:root:***** Running Evaluation *****
357
+ INFO:root: Num Turns = 7372
358
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
359
+ INFO:root:train size:8434, dev size:999, test size:1000
360
+ INFO:root:Generate setting:
361
+ use true_prev_bspn=False
362
+ use true_prev_aspn=False
363
+ use true_db_pointer=False
364
+ use true_prev_resp=False
365
+ use true_curr_bspn=False
366
+ use true_curr_aspn=False
367
+ use_all_previous_context=True
368
+ INFO:root:Running eval on test
369
+ INFO:root:***** Running Evaluation *****
370
+ INFO:root: Num Turns = 7372
371
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
372
+ INFO:root:train size:8434, dev size:999, test size:1000
373
+ INFO:root:Generate setting:
374
+ use true_prev_bspn=False
375
+ use true_prev_aspn=False
376
+ use true_db_pointer=False
377
+ use true_prev_resp=False
378
+ use true_curr_bspn=False
379
+ use true_curr_aspn=False
380
+ use_all_previous_context=True
381
+ INFO:root:Running eval on test
382
+ INFO:root:***** Running Evaluation *****
383
+ INFO:root: Num Turns = 7372
384
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
385
+ INFO:root:train size:8434, dev size:999, test size:1000
386
+ INFO:root:Generate setting:
387
+ use true_prev_bspn=False
388
+ use true_prev_aspn=False
389
+ use true_db_pointer=False
390
+ use true_prev_resp=False
391
+ use true_curr_bspn=False
392
+ use true_curr_aspn=False
393
+ use_all_previous_context=True
394
+ INFO:root:Running eval on test
395
+ INFO:root:***** Running Evaluation *****
396
+ INFO:root: Num Turns = 7372
397
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
398
+ INFO:root:train size:8434, dev size:999, test size:1000
399
+ INFO:root:Generate setting:
400
+ use true_prev_bspn=False
401
+ use true_prev_aspn=False
402
+ use true_db_pointer=False
403
+ use true_prev_resp=False
404
+ use true_curr_bspn=False
405
+ use true_curr_aspn=False
406
+ use_all_previous_context=True
407
+ INFO:root:Running eval on test
408
+ INFO:root:***** Running Evaluation *****
409
+ INFO:root: Num Turns = 7372
410
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
411
+ INFO:root:train size:8434, dev size:999, test size:1000
412
+ INFO:root:Generate setting:
413
+ use true_prev_bspn=False
414
+ use true_prev_aspn=False
415
+ use true_db_pointer=False
416
+ use true_prev_resp=False
417
+ use true_curr_bspn=False
418
+ use true_curr_aspn=False
419
+ use_all_previous_context=True
420
+ INFO:root:Running eval on test
421
+ INFO:root:***** Running Evaluation *****
422
+ INFO:root: Num Turns = 7372
423
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
424
+ INFO:root:train size:8434, dev size:999, test size:1000
425
+ INFO:root:Generate setting:
426
+ use true_prev_bspn=False
427
+ use true_prev_aspn=False
428
+ use true_db_pointer=False
429
+ use true_prev_resp=False
430
+ use true_curr_bspn=False
431
+ use true_curr_aspn=False
432
+ use_all_previous_context=True
433
+ INFO:root:Running eval on test
434
+ INFO:root:***** Running Evaluation *****
435
+ INFO:root: Num Turns = 7372
436
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
437
+ INFO:root:train size:8434, dev size:999, test size:1000
438
+ INFO:root:Generate setting:
439
+ use true_prev_bspn=False
440
+ use true_prev_aspn=False
441
+ use true_db_pointer=False
442
+ use true_prev_resp=False
443
+ use true_curr_bspn=False
444
+ use true_curr_aspn=False
445
+ use_all_previous_context=True
446
+ INFO:root:Running eval on test
447
+ INFO:root:***** Running Evaluation *****
448
+ INFO:root: Num Turns = 7372
449
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
450
+ INFO:root:train size:8434, dev size:999, test size:1000
451
+ INFO:root:Generate setting:
452
+ use true_prev_bspn=False
453
+ use true_prev_aspn=False
454
+ use true_db_pointer=False
455
+ use true_prev_resp=False
456
+ use true_curr_bspn=False
457
+ use true_curr_aspn=False
458
+ use_all_previous_context=True
459
+ INFO:root:Running eval on test
460
+ INFO:root:***** Running Evaluation *****
461
+ INFO:root: Num Turns = 7372
462
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
463
+ INFO:root:train size:8434, dev size:999, test size:1000
464
+ INFO:root:Generate setting:
465
+ use true_prev_bspn=False
466
+ use true_prev_aspn=False
467
+ use true_db_pointer=False
468
+ use true_prev_resp=False
469
+ use true_curr_bspn=False
470
+ use true_curr_aspn=False
471
+ use_all_previous_context=True
472
+ INFO:root:Running eval on test
473
+ INFO:root:***** Running Evaluation *****
474
+ INFO:root: Num Turns = 7372
475
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
476
+ INFO:root:train size:8434, dev size:999, test size:1000
477
+ INFO:root:Generate setting:
478
+ use true_prev_bspn=False
479
+ use true_prev_aspn=False
480
+ use true_db_pointer=False
481
+ use true_prev_resp=False
482
+ use true_curr_bspn=False
483
+ use true_curr_aspn=False
484
+ use_all_previous_context=True
485
+ INFO:root:Running eval on test
486
+ INFO:root:***** Running Evaluation *****
487
+ INFO:root: Num Turns = 7372
488
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
489
+ INFO:root:train size:8434, dev size:999, test size:1000
490
+ INFO:root:Generate setting:
491
+ use true_prev_bspn=False
492
+ use true_prev_aspn=False
493
+ use true_db_pointer=False
494
+ use true_prev_resp=False
495
+ use true_curr_bspn=False
496
+ use true_curr_aspn=False
497
+ use_all_previous_context=True
498
+ INFO:root:Running eval on test
499
+ INFO:root:***** Running Evaluation *****
500
+ INFO:root: Num Turns = 7372
501
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
502
+ INFO:root:train size:8434, dev size:999, test size:1000
503
+ INFO:root:Generate setting:
504
+ use true_prev_bspn=False
505
+ use true_prev_aspn=False
506
+ use true_db_pointer=False
507
+ use true_prev_resp=False
508
+ use true_curr_bspn=False
509
+ use true_curr_aspn=False
510
+ use_all_previous_context=True
511
+ INFO:root:Running eval on test
512
+ INFO:root:***** Running Evaluation *****
513
+ INFO:root: Num Turns = 7372
514
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
515
+ INFO:root:train size:8434, dev size:999, test size:1000
516
+ INFO:root:Generate setting:
517
+ use true_prev_bspn=False
518
+ use true_prev_aspn=False
519
+ use true_db_pointer=False
520
+ use true_prev_resp=False
521
+ use true_curr_bspn=False
522
+ use true_curr_aspn=False
523
+ use_all_previous_context=True
524
+ INFO:root:Running eval on test
525
+ INFO:root:***** Running Evaluation *****
526
+ INFO:root: Num Turns = 7372
527
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
528
+ INFO:root:train size:8434, dev size:999, test size:1000
529
+ INFO:root:Generate setting:
530
+ use true_prev_bspn=False
531
+ use true_prev_aspn=False
532
+ use true_db_pointer=False
533
+ use true_prev_resp=False
534
+ use true_curr_bspn=False
535
+ use true_curr_aspn=False
536
+ use_all_previous_context=True
537
+ INFO:root:Running eval on test
538
+ INFO:root:***** Running Evaluation *****
539
+ INFO:root: Num Turns = 7372
540
+ INFO:root:Reading encoded data from ./data/preprocessed/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
541
+ INFO:root:train size:8434, dev size:999, test size:1000
542
+ INFO:root:Generate setting:
543
+ use true_prev_bspn=False
544
+ use true_prev_aspn=False
545
+ use true_db_pointer=False
546
+ use true_prev_resp=False
547
+ use true_curr_bspn=False
548
+ use true_curr_aspn=False
549
+ use_all_previous_context=True
550
+ INFO:root:Running eval on test
551
+ INFO:root:***** Running Evaluation *****
552
+ INFO:root: Num Turns = 7372
553
+ INFO:root:Reading encoded data from ./data/preprocessed_gen_usr_utts/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
554
+ INFO:root:train size:14969, dev size:999, test size:1000
555
+ INFO:root:Generate setting:
556
+ use true_prev_bspn=False
557
+ use true_prev_aspn=False
558
+ use true_db_pointer=False
559
+ use true_prev_resp=False
560
+ use true_curr_bspn=False
561
+ use true_curr_aspn=False
562
+ use_all_previous_context=True
563
+ INFO:root:Running eval on test
564
+ INFO:root:***** Running Evaluation *****
565
+ INFO:root: Num Turns = 7372
566
+ INFO:root:Reading encoded data from ./data/preprocessed_gen_usr_utts/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
567
+ INFO:root:train size:14969, dev size:999, test size:1000
568
+ INFO:root:Generate setting:
569
+ use true_prev_bspn=False
570
+ use true_prev_aspn=False
571
+ use true_db_pointer=False
572
+ use true_prev_resp=False
573
+ use true_curr_bspn=False
574
+ use true_curr_aspn=False
575
+ use_all_previous_context=True
576
+ INFO:root:Running eval on test
577
+ INFO:root:Reading encoded data from ./data/preprocessed_gen_usr_utts/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
578
+ INFO:root:train size:14969, dev size:999, test size:1000
579
+ INFO:root:Generate setting:
580
+ use true_prev_bspn=False
581
+ use true_prev_aspn=False
582
+ use true_db_pointer=False
583
+ use true_prev_resp=False
584
+ use true_curr_bspn=False
585
+ use true_curr_aspn=False
586
+ use_all_previous_context=True
587
+ INFO:root:Running eval on test
588
+ INFO:root:***** Running Evaluation *****
589
+ INFO:root: Num Turns = 7372
590
+ INFO:root:Reading encoded data from ./data/preprocessed_gen_usr_utts/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
591
+ INFO:root:train size:14969, dev size:999, test size:1000
592
+ INFO:root:Generate setting:
593
+ use true_prev_bspn=False
594
+ use true_prev_aspn=False
595
+ use true_db_pointer=False
596
+ use true_prev_resp=False
597
+ use true_curr_bspn=False
598
+ use true_curr_aspn=False
599
+ use_all_previous_context=True
600
+ INFO:root:Running eval on test
601
+ INFO:root:***** Running Evaluation *****
602
+ INFO:root: Num Turns = 7372
603
+ INFO:root:inference time: 0.71 min
604
+ INFO:root:Reading encoded data from ./data/preprocessed_gen_usr_utts/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
605
+ INFO:root:train size:14969, dev size:999, test size:1000
606
+ INFO:root:Generate setting:
607
+ use true_prev_bspn=False
608
+ use true_prev_aspn=False
609
+ use true_db_pointer=False
610
+ use true_prev_resp=False
611
+ use true_curr_bspn=False
612
+ use true_curr_aspn=False
613
+ use_all_previous_context=True
614
+ INFO:root:Running eval on test
615
+ INFO:root:***** Running Evaluation *****
616
+ INFO:root: Num Turns = 7372
617
+ INFO:root:inference time: 0.89 min
618
+ INFO:root:Reading encoded data from ./data/preprocessed_gen_usr_utts/UBAR/multi-woz-processed/new_db_se_blank_encoded.data.json
619
+ INFO:root:train size:14969, dev size:999, test size:1000
620
+ INFO:root:Generate setting:
621
+ use true_prev_bspn=False
622
+ use true_prev_aspn=False
623
+ use true_db_pointer=False
624
+ use true_prev_resp=False
625
+ use true_curr_bspn=False
626
+ use true_curr_aspn=False
627
+ use_all_previous_context=True
628
+ INFO:root:Running eval on test
629
+ INFO:root:***** Running Evaluation *****
630
+ INFO:root: Num Turns = 7372
631
+ INFO:root:inference time: 0.10 min
632
+ INFO:root:Scoring time: 1.08 min
633
+ INFO:root:validation [CTR] match: 100.00 success: 100.00 bleu: 20.28 score: 120.28
634
+ INFO:root:update eval results to logs_test/UBAR-all_test.json
635
+ INFO:root:Evaluation finished