R0k1e
/

UltraLink-LM

 ---
 license: mit
+datasets:
+  - R0k1e/UltraLink
+  - stingning/ultrachat
+  - ise-uiuc/Magicoder-Evol-Instruct-110K
+  - ise-uiuc/Magicoder-OSS-Instruct-75K
+language:
+  - eng
+  - fra
+  - rus
+  - spa
+  - zho
+metrics:
+  - accuracy
 ---
+<img src="flow_diagram.png" alt="UltraLink Flow Diagram" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
+# Model Card for UltraLink-LM
+## Model Summary
+> The UltraLink-LM is a massively multilingual generative language model that follows instructions in 5 languages, English, French, Russian, Spanish, and Chinese. It is trained on a combination of publicly available datasets and UltraLink, including ShareGPT, UltraChat, Magicoder-Evol-Instruct-110K, and Magicoder-OSS-Instruct-75K. The model is capable of generating text in 5 languages with high quality and diversity.
+> UltraLink-LM outperforms [PolyLM-Chat-13b](https://huggingface.co/DAMO-NLP-MT/polylm-chat-13b), [Guanaco](JosephusCheung/Guanaco),  and [Bloomz-7b1-mt](https://huggingface.co/bigscience/bloomz-7b1-mt) in code, math and chat abilities in four languages, and has a high-quality and diverse text generation performance in all languages.
+> The UltraLink-LM is trained using [UltraLink](https://huggingface.co/datasets/R0k1e/UltraLink), [UltraChat](https://huggingface.co/datasets/stingning/ultrachat), [Magicoder-Evol](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K), [Magicoder-OSS](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K), and ShareGPT.
+> We release the checkpoints under a MIT license to further our mission of multilingual technologies empowering a multilingual world.
+- **Developed by:** [THUNLP]((http://nlp.csai.tsinghua.edu.cn/))
+- **Model type:** a Transformer style autoregressive massively multilingual language model.
+- **Paper**: [UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset](https://arxiv.org/abs/2402.04588)
+- **Languages**: Refer to the list of languages in the `language` section of this model card.
+- **License**: MIT
+- **Model**: [UltraLink-LM](https://huggingface.co/R0k1e/UltraLink-LM)
+- **Model Size**: 13 billion parameters
+- **Datasets**: [UltraLink](https://huggingface.co/datasets/R0k1e/UltraLink), [UltraChat](https://huggingface.co/datasets/stingning/ultrachat), [Magicoder-Evol](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K), [Magicoder-OSS](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K), and ShareGPT.
+## Use
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+checkpoint = "R0k1e/UltraLink-LM"
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+ultralink_lm = AutoModelForCausalLM.from_pretrained(checkpoint)
+# Chat abilities in Chinese
+# Please tell us about Tang San Cai(An ancient Chinese pottery type).
+chat_inputs = tokenizer.encode("请介绍一下唐三彩。", return_tensors="pt")
+chat_outputs = ultralink_lm.generate(chat_inputs, max_new_tokens=512)
+print(tokenizer.decode(chat_outputs[0]))
+# Expected output:
+"""
+唐三彩是中国古代的一种绘画艺术形式，起源于唐朝（618-907）。它以其精美的色彩和细致的绘画技巧而闻名。唐三彩是一种陶瓷，通常以圆形或扁平的形状制成，并刻有图案或图像。这些图案通常是以彩色的方式呈现，使用釉质陶瓷上的颜料。唐三彩的主题各异，包括动物、植物、人物和日常生活场景。它是中国艺朩的重要贡献，仍然是中国艺术的重要组成部分。
+"""
+# Translations in English:
+"""
+Tang Sancai is an ancient Chinese art form of painting that originated in the Tang Dynasty (618-907). It is known for its exquisite colours and detailed painting techniques. Tang Sancai is a type of ceramic, usually made in round or flat shapes and inscribed with designs or images. These designs are usually rendered in colour, using pigments on glazed ceramics. Tang sancai has a variety of themes, including animals, plants, figures and scenes from daily life. It was an important contribution to Chinese art kwun and remains an important part of Chinese art.
+"""
+# Code abilities in Russian
+# Please implement a bubble sort algorithm in Python.
+code_inputs = tokenizer.encode("Реализуйте алгоритм пузырьковой сортировки на Python.", return_tensors="pt")
+code_outputs = ultralink_lm.generate(code_inputs, max_new_tokens=512)
+print(tokenizer.decode(code_outputs[0]))
+# Expected output:
+"""
+Вот простая реализация алгоритма пузырьковой сортировки на Python:
+```python
+def bubbleSort(arr):
+    n = len(arr)
+    for i in range(n):
+        for j in range(0, n-i-1):
+            if arr[j] > arr[j+1]:
+                arr[j], arr[j+1] = arr[j+1], arr[j]
+arr = [64, 34, 25, 12, 22, 11, 90]
+bubbleSort(arr)
+print("Отсортированный массив:", arr)
+\```
+В этом коде функция `bubbleSort` сортирует массив `arr` с помощью алгоритма пузырьковой сортировки. Внутри функции происходит итерация по массиву, сравнение соседних элементов и обмен их местами, если они находятся в неправильном порядке. Этот процесс повторяется до тех пор, пока массив не будет отсортирован.
+После вызова функции `bubbleSort` отсортированный массив выводится на экран.
+Обратите внимание, что алгоритм пузырьковой сортировки не является эффективным для больших наборов данных, так как его средняя и худшая временная сложность составляют O(n^2), где n - количество элементов в массиве.
+"""
+# Translations in English:
+"""
+Here is a simple implementation of the bubble sort algorithm in Python:
+```python
+def bubbleSort(arr):
+    n = len(arr)
+    for i in range(n):
+        for j in range(0, n-i-1):
+            if arr[j] > arr[j+1]:
+                arr[j], arr[j+1] = arr[j+1], arr[j]
+arr = [64, 34, 25, 12, 22, 11, 90]
+bubbleSort(arr)
+print("Sorted array:", arr)
+\```
+In this code, the `bubbleSort` function sorts the `arr` array using the bubble sort algorithm. Inside the function iterates through the array, comparing neighbouring elements and swapping them if they are in the wrong order. This process is repeated until the array is sorted.
+After calling the `bubbleSort` function, the sorted array is displayed on the screen.
+Note that the bubble sort algorithm is not efficient for large datasets because its average and worst-case time complexity are O(n^2), where n is the number of elements in the array.
+"""
+# Math abilities in French
+# When the length of a rectangle is twice its width, calculate the area of the rectangle if its perimeter is 18 units?
+math_inputs = tokenizer.encode("Lorsque la longueur d'un rectangle est le double de sa largeur, calculer l'aire du rectangle si son périmètre est de 18 unités?", return_tensors="pt")
+math_outputs = ultralink_lm.generate(math_inputs, max_new_tokens=512)
+print(tokenizer.decode(math_outputs[0]))
+# Expected output:
+"""
+La longueur du rectangle est deux fois sa largeur, donc la longueur est de 2w et la largeur est de w.
+Le périmètre d'un rectangle est deux fois la somme de sa longueur et de sa largeur, donc l'équation est 2(2w + w) = 18.
+En simplifiant l'équation, nous obtenons 2(3w) = 18, ce qui se simplifie en 6w = 18.
+En divisant les deux côtés par 6, nous obtenons w = 3.
+Par conséquent, la longueur du rectangle est de 2w = 2(3) = 6.
+L'aire d'un rectangle est le produit de sa longueur et de sa largeur, donc l'aire est de 6 * 3 = 18.
+La réponse est : 18
+"""
+# Translations in English:
+"""
+The length of the rectangle is twice its width, so the length is 2w and the width is w.
+The perimeter of a rectangle is twice the sum of its length and width, so the equation is 2(2w + w) = 18.
+Simplifying the equation, we get 2(3w) = 18, which simplifies to 6w = 18.
+Dividing the two sides by 6 gives w = 3.
+So the length of the rectangle is 2w = 2(3) = 6.
+The area of a rectangle is the product of its length and width, so the area is 6 * 3 = 18.
+The answer is: 18
+"""
+```
+## Model Details
+### Finetuning
+- Architecture: Same as [Llama-2-13b](https://huggingface.co/meta-llama/Llama-2-13b-hf)
+- Number of Samples seen during Finetuning: 1023K
+- Batch size: 128
+- Hardware: NVIDIA A100 80GB PCIe
+- Software: BMTrain
+### Data Sources
+The UltraLink-LM is trained on the following datasets:
+- [UltraLink](https://huggingface.co/datasets/R0k1e/UltraLink)
+- [UltraChat](https://huggingface.co/datasets/stingning/ultrachat)
+- [Magicoder-Evol](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K)
+- [Magicoder-OSS](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K)
+- ShareGPT
+All the datasets are integrated into the UltraLink dataset.
+## Evaluation
+### Multilingual HumanEval
+[HumanEval](https://github.com/openai/human-eval) is a well-known benchmark for evaluating the code ability of LLMs. It execute the code snippets generated by the model and evaluate their correctness.  Since there are no existing multilingual test set for code generation, we use GPT-3.5 with carefully-designed prompts to translation HumanEval into other languages.
+|Model|En|Zh|Es|Ru|Fr|Avg|
+|-----|---|---|---|---|---|---|
+|Bloomz-7b1-mt | 8.5 | 7.3 | 6.1 | 8.5 | 6.1 | 7.3 |
+|Phoenix-inst-chat-7b | 11.0 | 10.4 | 8.5 | 1.2 | 13.4 | 12.2 |
+|PolyLM-Multialpaca-13b | 8.5 | 7.3 | 6.1 | 6.1 | 6.1 | 6.8 |
+|PolyLM-Chat-13b | 10.4 | 7.9 | 6.1 | 7.3 | 8.5 | 8.1 |
+|Chimera-inst-chat-13b| 14.6 | 13.4 | 14.6 | 12.8 | 14.0 | 13.9 |
+|Okapi-7b | 12.2 | 11.0 | 8.5 | 8.5 | 8.5 | 9.8 |
+|Guanaco-7b | 9.2 | 6.7 | 11.0 | 9.8 | 12.8 | 9.9 |
+|Guanaco-13b| 18.3 | 15.9 | 9.8 | 8.5 | 14.6 | 12.2 |
+|UltraLink-LM  | 60.4 | 43.9 | 40.9 | 49.4 | 39.6 | 46.8|
+### MGSM
+We employ [MGSM](https://github.com/google-research/url-nlp/tree/main/mgsm) to evaluate the math reasoning abilities, which is a multilingual benchmark. It compares the result with correct answers and evaluates the model's ability to perform mathematical reasoning.
+|Model|En|Zh|Es|Ru|Fr|Avg|
+|-----|---|---|---|---|---|---|
+|Bloomz-7b1-mt | 2.8 | 1.6 | 2.0 | 0.4 | 2.8 | 1.7 |
+|Phoenix-inst-chat-7b | 3.2 | 3.2 | 2.8 | 3.2 | 3.2 | 3.1 |
+|PolyLM-Multialpaca-13b | 1.2 | 2.8 | 1.6 | 2.8 | 2.4 | 2.4 |
+|PolyLM-Chat-13b | 10.8 | 6.4 | 4.8 | 4.4 | 5.6 | 5.3 |
+|Chimera-inst-chat-13b  | 14.0 | 11.6 | 10.0 | 12.0 | 12.8 | 11.6 |
+|Okapi-7b | 4.0 | 2.4 | 3.6 | 4.4 | 4.8 | 3.8 |
+|Guanaco-7b | 4.0 | 1.6 | 3.2 | 2.8 | 4.4 | 3.0 |
+|Guanaco-13b | 13.6 | 10.8 | 11.2 | 6.4 | 5.2 | 8.4 |
+|UltraLink-LM| 70.4 | 56.0 | 70.4 | 64.8 | 63.6 | 63.7  |
+### OMGEval
+We use the [OMGEval](https://github.com/blcuicall/OMGEval) to evaluate the chat ability, which is a multilingual version of the widely-used English benchmark AlpacaEval.
+|Model|En|Zh|Es|Ru|Fr|Avg|
+|-----|---|---|---|---|---|---|
+|Bloomz-7b1-mt | 0.0 | 0.9 | 0.1 | 0.5 | 0.3 | 0.4 |
+|Phoenix-inst-chat-7b  | 6.9 | 13.3 | 7.4 | 2.9 | 8.1 | 7.7 |
+|PolyLM-Multialpaca-13b  | 3.4 | 5.0 | 2.1 | 5.1 | 2.2 | 3.6 |
+|PolyLM-Chat-13b | 7.7 | 14.0 | 6.1 | 5.5 | 4.8 | 7.6 |
+|Chimera-inst-chat-13b | 15.5 | 9.7 | 11.8 | 13.7 | 13.8 | 12.9 |
+|Okapi-7b | 8.8 | 6.2 | 5.0 | 12.1 | 8.7 | 8.2 |
+|Guanaco-7b  | 4.6 | 3.8 | 0.4 | 1.8 | 1.2 | 2.4 |
+|Guanaco-13b  |  29.0 | 8.6 | 16.9 | 15.4 | 17.3 | 17.5 |
+|UltraLink-LM |  28.8 |  21.9 |  23.5 |  37.6 |  29.0 |  28.2  |
+## Citation
+```bibtex
+@misc{wang2024ultralink,
+      title={UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset},
+      author={Haoyu Wang and Shuo Wang and Yukun Yan and Xujia Wang and Zhiyu Yang and Yuzhuang Xu and Zhenghao Liu and Ning Ding and Xu Han and Zhiyuan Liu and Maosong Sun},
+      year={2024},
+      eprint={2402.04588},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```

flow_diagram.png ADDED Viewed

infer.py ADDED Viewed

	@@ -0,0 +1,114 @@

+from transformers import AutoModelForCausalLM, AutoTokenizer
+checkpoint = "R0k1e/UltraLink-LM"
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+ultralink_lm = AutoModelForCausalLM.from_pretrained(checkpoint)
+# Chat abilities in Chinese
+# Please tell us about Tang San Cai(An ancient Chinese pottery type).
+chat_inputs = tokenizer.encode("请介绍一下唐三彩。", return_tensors="pt")
+chat_outputs = ultralink_lm.generate(chat_inputs, max_new_tokens=512)
+print(tokenizer.decode(chat_outputs[0]))
+# Expected output:
+"""
+唐三彩是中国古代的一种绘画艺术形式，起源于唐朝（618-907）。它以其精美的色彩和细致的绘画技巧而闻名。唐三彩是一种陶瓷，通常以圆形或扁平的形状制成，并刻有图案或图像。这些图案通常是以彩色的方式呈现，使用釉质陶瓷上的颜料。唐三彩的主题各异，包括动物、植物、人物和日常生活场景。它是中国艺朩的重要贡献，仍然是中国艺术的重要组成部分。
+"""
+# Translations in English:
+"""
+Tang Sancai is an ancient Chinese art form of painting that originated in the Tang Dynasty (618-907). It is known for its exquisite colours and detailed painting techniques. Tang Sancai is a type of ceramic, usually made in round or flat shapes and inscribed with designs or images. These designs are usually rendered in colour, using pigments on glazed ceramics. Tang sancai has a variety of themes, including animals, plants, figures and scenes from daily life. It was an important contribution to Chinese art kwun and remains an important part of Chinese art.
+"""
+# Code abilities in Russian
+# Please implement a bubble sort algorithm in Python.
+code_inputs = tokenizer.encode("Реализуйте алгоритм пузырьковой сортировки на Python.", return_tensors="pt")
+code_outputs = ultralink_lm.generate(code_inputs, max_new_tokens=512)
+print(tokenizer.decode(code_outputs[0]))
+# Expected output:
+"""
+Вот простая реализация алгоритма пузырьковой сортировки на Python:
+```python
+def bubbleSort(arr):
+    n = len(arr)
+    for i in range(n):
+        for j in range(0, n-i-1):
+            if arr[j] > arr[j+1]:
+                arr[j], arr[j+1] = arr[j+1], arr[j]
+arr = [64, 34, 25, 12, 22, 11, 90]
+bubbleSort(arr)
+print("Отсортированный массив:", arr)
+```
+В этом коде функция `bubbleSort` сортирует массив `arr` с помощью алгоритма пузырьковой сортировки. Внутри функции происходит итерация по массиву, сравнение соседних элементов и обмен их местами, если они находятся в неправильном порядке. Этот процесс повторяется до тех пор, пока массив не будет отсортирован.
+После вызова функции `bubbleSort` отсортированный массив выводится на экран.
+Обратите внимание, что алгоритм пузырьковой сортировки не является эффективным для больших наборов данных, так как его средняя и худшая временная сложность составляют O(n^2), где n - количество элементов в массиве.
+"""
+# Translations in English:
+"""
+Here is a simple implementation of the bubble sort algorithm in Python:
+```python
+def bubbleSort(arr):
+    n = len(arr)
+    for i in range(n):
+        for j in range(0, n-i-1):
+            if arr[j] > arr[j+1]:
+                arr[j], arr[j+1] = arr[j+1], arr[j]
+arr = [64, 34, 25, 12, 22, 11, 90]
+bubbleSort(arr)
+print("Sorted array:", arr)
+```
+In this code, the `bubbleSort` function sorts the `arr` array using the bubble sort algorithm. Inside the function iterates through the array, comparing neighbouring elements and swapping them if they are in the wrong order. This process is repeated until the array is sorted.
+After calling the `bubbleSort` function, the sorted array is displayed on the screen.
+Note that the bubble sort algorithm is not efficient for large datasets because its average and worst-case time complexity are O(n^2), where n is the number of elements in the array.
+"""
+# Math abilities in French
+# When the length of a rectangle is twice its width, calculate the area of the rectangle if its perimeter is 18 units?
+math_inputs = tokenizer.encode("Lorsque la longueur d'un rectangle est le double de sa largeur, calculer l'aire du rectangle si son périmètre est de 18 unités?", return_tensors="pt")
+math_outputs = ultralink_lm.generate(math_inputs, max_new_tokens=512)
+print(tokenizer.decode(math_outputs[0]))
+# Expected output:
+"""
+La longueur du rectangle est deux fois sa largeur, donc la longueur est de 2w et la largeur est de w.
+Le périmètre d'un rectangle est deux fois la somme de sa longueur et de sa largeur, donc l'équation est 2(2w + w) = 18.
+En simplifiant l'équation, nous obtenons 2(3w) = 18, ce qui se simplifie en 6w = 18.
+En divisant les deux côtés par 6, nous obtenons w = 3.
+Par conséquent, la longueur du rectangle est de 2w = 2(3) = 6.
+L'aire d'un rectangle est le produit de sa longueur et de sa largeur, donc l'aire est de 6 * 3 = 18.
+La réponse est : 18
+"""
+# Translations in English:
+"""
+The length of the rectangle is twice its width, so the length is 2w and the width is w.
+The perimeter of a rectangle is twice the sum of its length and width, so the equation is 2(2w + w) = 18.
+Simplifying the equation, we get 2(3w) = 18, which simplifies to 6w = 18.
+Dividing the two sides by 6 gives w = 3.
+So the length of the rectangle is 2w = 2(3) = 6.
+The area of a rectangle is the product of its length and width, so the area is 6 * 3 = 18.
+The answer is: 18
+"""