Hugging Face's logo

sinequa
/
passage-ranker.mango

Text Classification
Transformers
PyTorch
9 languages
bert
Inference Endpoints
Model card Files Community
5
youval commited on
Commit
028bd3e
•
1 Parent(s): 67188de

Update model card

Browse files
Files changed (1) hide show
  1. README.md +124 -119
README.md CHANGED
@@ -1,119 +1,124 @@
1
- ---
2
- language:
3
- - de
4
- - en
5
- - es
6
- - fr
7
- - it
8
- - ja
9
- - nl
10
- - pt
11
- - zh
12
- ---
13
-
14
- # Model Card for `passage-ranker.mango`
15
-
16
- This model is a passage ranker developed by Sinequa. It produces a relevance score given a query-passage pair and is
17
- used to order search results.
18
-
19
- Model name: `passage-ranker.mango`
20
-
21
- ## Supported Languages
22
-
23
- The model was trained and tested in the following languages:
24
-
25
- - Chinese (simplified)
26
- - Dutch
27
- - English
28
- - French
29
- - German
30
- - Italian
31
- - Japanese
32
- - Portuguese
33
- - Spanish
34
-
35
- Besides the aforementioned languages, basic support can be expected for additional 93 languages that were used during
36
- the pretraining of the base model (see
37
- [list of languages](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages)).
38
-
39
- ## Scores
40
-
41
- | Metric | Value |
42
- |:--------------------|------:|
43
- | Relevance (NDCG@10) | 0.480 |
44
-
45
- Note that the relevance score is computed as an average over 14 retrieval datasets (see
46
- [details below](#evaluation-metrics)).
47
-
48
- ## Inference Times
49
-
50
- | GPU | Batch size 32 |
51
- |:-----------|--------------:|
52
- | NVIDIA A10 | 84 ms |
53
- | NVIDIA T4 | 358 ms |
54
-
55
- The inference times only measure the time the model takes to process a single batch, it does not include pre- or
56
- post-processing steps like the tokenization.
57
-
58
- ## Requirements
59
-
60
- - Minimal Sinequa version: 11.10.0
61
- - GPU memory usage: 1070 MiB
62
-
63
- Note that GPU memory usage only includes how much GPU memory the actual model consumes on an NVIDIA T4 GPU with a batch
64
- size of 32. It does not include the fix amount of memory that is consumed by the ONNX Runtime upon initialization which
65
- can be around 0.5 to 1 GiB depending on the used GPU.
66
-
67
- ## Model Details
68
-
69
- ### Overview
70
-
71
- - Number of parameters: 167 million
72
- - Base language model: [Multilingual BERT-Base](https://huggingface.co/bert-base-multilingual-uncased)
73
- - Insensitive to casing and accents
74
- - Training procedure: [MonoBERT](https://arxiv.org/abs/1901.04085)
75
-
76
- ### Training Data
77
-
78
- - MS MARCO Passage Ranking
79
- ([Paper](https://arxiv.org/abs/1611.09268),
80
- [Official Page](https://microsoft.github.io/msmarco/),
81
- [English & translated datasets on the HF dataset hub](https://huggingface.co/datasets/unicamp-dl/mmarco))
82
- - Original English dataset
83
- - Translated datasets for the other eight supported languages
84
-
85
- ### Evaluation Metrics
86
-
87
- To determine the relevance score, we averaged the results that we obtained when evaluating on the datasets of the
88
- [BEIR benchmark](https://github.com/beir-cellar/beir). Note that all these datasets are in English.
89
-
90
- | Dataset | NDCG@10 |
91
- |:------------------|--------:|
92
- | Average | 0.480 |
93
- | | |
94
- | Arguana | 0.537 |
95
- | CLIMATE-FEVER | 0.241 |
96
- | DBPedia Entity | 0.371 |
97
- | FEVER | 0.777 |
98
- | FiQA-2018 | 0.327 |
99
- | HotpotQA | 0.696 |
100
- | MS MARCO | 0.414 |
101
- | NFCorpus | 0.332 |
102
- | NQ | 0.484 |
103
- | Quora | 0.768 |
104
- | SCIDOCS | 0.143 |
105
- | SciFact | 0.648 |
106
- | TREC-COVID | 0.673 |
107
- | Webis-Touche-2020 | 0.310 |
108
-
109
- We evaluated the model on the datasets of the [MIRACL benchmark](https://github.com/project-miracl/miracl) to test its
110
- multilingual capacities. Note that not all training languages are part of the benchmark, so we only report the metrics
111
- for the existing languages.
112
-
113
- | Language | NDCG@10 |
114
- |:----------------------|--------:|
115
- | Chinese (simplified) | 0.463 |
116
- | French | 0.447 |
117
- | German | 0.415 |
118
- | Japanese | 0.526 |
119
- | Spanish | 0.485 |
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - de
4
+ - en
5
+ - es
6
+ - fr
7
+ - it
8
+ - ja
9
+ - nl
10
+ - pt
11
+ - zh
12
+ ---
13
+
14
+ # Model Card for `passage-ranker.mango`
15
+
16
+ This model is a passage ranker developed by Sinequa. It produces a relevance score given a query-passage pair and is used to order search results.
17
+
18
+ Model name: `passage-ranker.mango`
19
+
20
+ ## Supported Languages
21
+
22
+ The model was trained and tested in the following languages:
23
+
24
+ - Chinese (simplified)
25
+ - Dutch
26
+ - English
27
+ - French
28
+ - German
29
+ - Italian
30
+ - Japanese
31
+ - Portuguese
32
+ - Spanish
33
+
34
+ Besides the aforementioned languages, basic support can be expected for additional 93 languages that were used during the pretraining of the base model (see
35
+ [list of languages](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages)).
36
+
37
+ ## Scores
38
+
39
+ | Metric | Value |
40
+ |:--------------------|------:|
41
+ | Relevance (NDCG@10) | 0.480 |
42
+
43
+ Note that the relevance score is computed as an average over 14 retrieval datasets (see
44
+ [details below](#evaluation-metrics)).
45
+
46
+ ## Inference Times
47
+
48
+ | GPU | Quantization type | Batch size 1 | Batch size 32 |
49
+ |:------------------------------------------|:------------------|---------------:|---------------:|
50
+ | NVIDIA A10 | FP16 | 2 ms | 28 ms |
51
+ | NVIDIA A10 | FP32 | 4 ms | 82 ms |
52
+ | NVIDIA T4 | FP16 | 3 ms | 65 ms |
53
+ | NVIDIA T4 | FP32 | 14 ms | 369 ms |
54
+ | NVIDIA L4 | FP16 | 3 ms | 38 ms |
55
+ | NVIDIA L4 | FP32 | 5 ms | 123 ms |
56
+
57
+ ## Gpu Memory usage
58
+
59
+ | Quantization type | Memory |
60
+ |:-------------------------------------------------|-----------:|
61
+ | FP16 | 850 MiB |
62
+ | FP32 | 1200 MiB |
63
+
64
+ Note that GPU memory usage only includes how much GPU memory the actual model consumes on an NVIDIA T4 GPU with a batch
65
+ size of 32. It does not include the fix amount of memory that is consumed by the ONNX Runtime upon initialization which
66
+ can be around 0.5 to 1 GiB depending on the used GPU.
67
+
68
+ ## Requirements
69
+
70
+ - Minimal Sinequa version: 11.10.0
71
+ - Minimal Sinequa version for using FP16 models and GPUs with CUDA compute capability of 8.9+ (like NVIDIA L4): 11.11.0
72
+ - [Cuda compute capability](https://developer.nvidia.com/cuda-gpus): above 5.0 (above 6.0 for FP16 use)
73
+
74
+ ## Model Details
75
+
76
+ ### Overview
77
+
78
+ - Number of parameters: 167 million
79
+ - Base language model: [Multilingual BERT-Base](https://huggingface.co/bert-base-multilingual-uncased)
80
+ - Insensitive to casing and accents
81
+ - Training procedure: [MonoBERT](https://arxiv.org/abs/1901.04085)
82
+
83
+ ### Training Data
84
+
85
+ - MS MARCO Passage Ranking
86
+ ([Paper](https://arxiv.org/abs/1611.09268),
87
+ [Official Page](https://microsoft.github.io/msmarco/),
88
+ [English & translated datasets on the HF dataset hub](https://huggingface.co/datasets/unicamp-dl/mmarco))
89
+ - Original English dataset
90
+ - Translated datasets for the other eight supported languages
91
+
92
+ ### Evaluation Metrics
93
+
94
+ To determine the relevance score, we averaged the results that we obtained when evaluating on the datasets of the
95
+ [BEIR benchmark](https://github.com/beir-cellar/beir). Note that all these datasets are in English.
96
+
97
+ | Dataset | NDCG@10 |
98
+ |:------------------|--------:|
99
+ | Average | 0.480 |
100
+ | | |
101
+ | Arguana | 0.537 |
102
+ | CLIMATE-FEVER | 0.241 |
103
+ | DBPedia Entity | 0.371 |
104
+ | FEVER | 0.777 |
105
+ | FiQA-2018 | 0.327 |
106
+ | HotpotQA | 0.696 |
107
+ | MS MARCO | 0.414 |
108
+ | NFCorpus | 0.332 |
109
+ | NQ | 0.484 |
110
+ | Quora | 0.768 |
111
+ | SCIDOCS | 0.143 |
112
+ | SciFact | 0.648 |
113
+ | TREC-COVID | 0.673 |
114
+ | Webis-Touche-2020 | 0.310 |
115
+
116
+ We evaluated the model on the datasets of the [MIRACL benchmark](https://github.com/project-miracl/miracl) to test its multilingual capacities. Note that not all training languages are part of the benchmark, so we only report the metrics for the existing languages.
117
+
118
+ | Language | NDCG@10 |
119
+ |:----------------------|--------:|
120
+ | Chinese (simplified) | 0.463 |
121
+ | French | 0.447 |
122
+ | German | 0.415 |
123
+ | Japanese | 0.526 |
124
+ | Spanish | 0.485 |