Update README.md

README.md

@@ -24,8 +24,8 @@ language:
 - **License(s):** [llama3.1](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/blob/main/LICENSE)
 - **Model Developers:** Neural Magic
 
-Quantized version of [Meta-Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct). It achieves an average recovery of 99.81% on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1) compared to the unquantized model.
-<!-- It achieves an average score of 77.75 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 78.67. -->
+Quantized version of [Meta-Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct).
+It achieves an average score of 86.41 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 86.63.
 
 ### Model Optimizations
 
@@ -162,7 +162,8 @@ oneshot(
 
 ## Evaluation
 
-The model was evaluated on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) leaderboard tasks (version 1) with the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and the [vLLM](https://docs.vllm.ai/en/stable/) engine, using the following command:
+The model was evaluated on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) leaderboard tasks (version 1) with the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and the [vLLM](https://docs.vllm.ai/en/stable/) engine, using the following command.
+A modified version of ARC-C and GSM8k-cot was used for evaluations, in line with Llama 3.1's prompting. It can be accessed on the [Neural Magic fork of the lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness/tree/llama_3.1_instruct).
 ```
 lm_eval \
   --model vllm \
@@ -170,7 +171,6 @@ lm_eval \
   --tasks openllm \
   --batch_size auto
 ```
-Certain benchmarks for the full precision model are still being acquired. Average recovery is calculated only with metrics that both models have been evaluated on.
 
 ### Accuracy
 
@@ -189,41 +189,41 @@ Certain benchmarks for the full precision model are still being acquired. Averag
   <tr>
    <td>MMLU (5-shot)
    </td>
-   <td>*
+   <td>86.25
    </td>
    <td>86.06
    </td>
-   <td>*
+   <td>99.78%
    </td>
   </tr>
   <tr>
-   <td>ARC Challenge (25-shot)
+   <td>ARC Challenge (0-shot)
    </td>
-   <td>*
+   <td>96.93
    </td>
-   <td>*
+   <td>96.33
    </td>
-   <td>*
+   <td>99.38%
    </td>
   </tr>
   <tr>
-   <td>GSM-8K (5-shot, strict-match)
+   <td>GSM-8K-cot (8-shot, strict-match)
    </td>
-   <td>95.07
+   <td>96.44
    </td>
-   <td>94.39
+   <td>95.91
    </td>
-   <td>99.28%
+   <td>99.45%
    </td>
   </tr>
   <tr>
    <td>Hellaswag (10-shot)
    </td>
-   <td>*
+   <td>88.33
    </td>
    <td>88.25
    </td>
-   <td>*
+   <td>99.91%
    </td>
   </tr>
   <tr>
@@ -249,11 +249,11 @@ Certain benchmarks for the full precision model are still being acquired. Averag
   <tr>
    <td><strong>Average</strong>
    </td>
-   <td><strong>*</strong>
+   <td><strong>86.63</strong>
    </td>
-   <td><strong>*</strong>
+   <td><strong>86.41</strong>
    </td>
-   <td><strong>99.81%</strong>
+   <td><strong>99.74%</strong>
    </td>
   </tr>
 </table>