JeongwonChoi
commited on
Commit
β’
0325e40
1
Parent(s):
060ff4f
Update README.md
Browse files
README.md
CHANGED
@@ -1,84 +1,101 @@
|
|
1 |
---
|
2 |
tags:
|
3 |
-
- text-generation
|
4 |
license: cc-by-nc-sa-4.0
|
5 |
language:
|
6 |
-
- ko
|
7 |
base_model: mistralai/Mistral-7B-Instruct-v0.2
|
8 |
pipeline_tag: text-generation
|
|
|
|
|
9 |
---
|
10 |
|
11 |
# **DataVortexM-7B-Instruct-v0.1**
|
12 |
-
<img src="https://imgur.com/lpXTyPe.png" alt="DataVortex" style="height: 8em;">
|
13 |
|
14 |
-
|
15 |
-
|
16 |
-
[cc-by-nc-sa-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
|
17 |
|
18 |
## **Model Details**
|
19 |
|
20 |
### **Base Model**
|
21 |
-
|
|
|
22 |
|
23 |
### **Trained On**
|
24 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
25 |
|
26 |
### **Instruction format**
|
27 |
|
28 |
It follows **Alpaca** format.
|
29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
30 |
## **Model Benchmark**
|
31 |
|
32 |
### **Ko-LLM-Leaderboard**
|
33 |
|
34 |
-
-
|
35 |
-
|
36 |
-
-
|
37 |
-
- **Ko-MMLU**: 38.73
|
38 |
-
- **Ko-TruthfulQA**: 45.46
|
39 |
-
- **Ko-CommonGen V2**: 38.37
|
40 |
|
41 |
-
|
42 |
|
43 |
-
|
44 |
You can use the code below.
|
45 |
|
46 |
```python
|
47 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
48 |
|
49 |
-
device = "cuda"
|
50 |
|
51 |
-
model = AutoModelForCausalLM.from_pretrained("
|
52 |
-
tokenizer = AutoTokenizer.from_pretrained("
|
53 |
|
54 |
messages = [
|
55 |
-
{
|
|
|
|
|
|
|
56 |
]
|
57 |
|
58 |
-
|
59 |
-
|
60 |
-
|
61 |
-
|
62 |
-
|
63 |
-
|
64 |
-
|
65 |
-
decoded
|
66 |
-
input_ids=encoded,
|
67 |
-
temperature=0.2,
|
68 |
-
top_p=0.9,
|
69 |
-
repetition_penalty=1.2,
|
70 |
-
do_sample=True,
|
71 |
-
max_length=4096,
|
72 |
-
eos_token_id=tokenizer.eos_token_id,
|
73 |
-
pad_token_id=tokenizer.eos_token_id
|
74 |
-
)
|
75 |
-
decoded = decoded[0][encoded.shape[1]:decoded[0].shape[-1]]
|
76 |
-
decoded_text = tokenizer.decode(decoded, skip_special_tokens=True)
|
77 |
-
print(decoded_text)
|
78 |
```
|
79 |
|
|
|
|
|
|
|
|
|
80 |
<div align="center">
|
81 |
<a href="https://edentns.com/">
|
82 |
-
<img src="
|
83 |
</a>
|
84 |
</div>
|
|
|
1 |
---
|
2 |
tags:
|
3 |
+
- text-generation
|
4 |
license: cc-by-nc-sa-4.0
|
5 |
language:
|
6 |
+
- ko
|
7 |
base_model: mistralai/Mistral-7B-Instruct-v0.2
|
8 |
pipeline_tag: text-generation
|
9 |
+
datasets:
|
10 |
+
- beomi/KoAlpaca-v1.1a
|
11 |
---
|
12 |
|
13 |
# **DataVortexM-7B-Instruct-v0.1**
|
|
|
14 |
|
15 |
+
<img src="./DataVortex.png" alt="DataVortex" style="height: 8em;">
|
|
|
|
|
16 |
|
17 |
## **Model Details**
|
18 |
|
19 |
### **Base Model**
|
20 |
+
|
21 |
+
[mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)
|
22 |
|
23 |
### **Trained On**
|
24 |
+
|
25 |
+
- **OS**: Ubuntu 20.04
|
26 |
+
- **GPU**: H100 80GB x4
|
27 |
+
- **transformers**: v4.36.2
|
28 |
+
|
29 |
+
### **Dataset**
|
30 |
+
|
31 |
+
- [beomi/KoAlpaca-v1.1a](https://huggingface.co/datasets/beomi/KoAlpaca-v1.1a) - 21k rows
|
32 |
|
33 |
### **Instruction format**
|
34 |
|
35 |
It follows **Alpaca** format.
|
36 |
|
37 |
+
E.g.
|
38 |
+
|
39 |
+
```python
|
40 |
+
text = """\
|
41 |
+
### System:
|
42 |
+
λΉμ μ μ¬λλ€μ΄ μ 보λ₯Ό μ°Ύμ μ μλλ‘ λμμ£Όλ μΈκ³΅μ§λ₯ λΉμμ
λλ€.
|
43 |
+
|
44 |
+
### User:
|
45 |
+
λνλ―Όκ΅μ μλλ μ΄λμΌ?
|
46 |
+
|
47 |
+
### Assistant:
|
48 |
+
λνλ―Όκ΅μ μλλ μμΈμ
λλ€.
|
49 |
+
|
50 |
+
### User:
|
51 |
+
μμΈ μΈκ΅¬λ μ΄ λͺ λͺ
μ΄μΌ?
|
52 |
+
"""
|
53 |
+
```
|
54 |
+
|
55 |
## **Model Benchmark**
|
56 |
|
57 |
### **Ko-LLM-Leaderboard**
|
58 |
|
59 |
+
| Model | Average | Ko-ARC | Ko-HellaSwag | Ko-MMLU | Ko-TruthfulQA | Ko-CommonGen V2 |
|
60 |
+
| -------------------------------- | --------- | --------- | ------------ | --------- | ------------- | --------------- |
|
61 |
+
| **DataVortexM-7B-Instruct-v0.1** | **39.81** | **34.13** | **42.35** | **38.73** | **45.46** | **38.37** |
|
|
|
|
|
|
|
62 |
|
63 |
+
## **Implementation Code**
|
64 |
|
65 |
+
This model contains the chat_template instruction format.
|
66 |
You can use the code below.
|
67 |
|
68 |
```python
|
69 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
70 |
|
71 |
+
device = "cuda" # the device to load the model onto
|
72 |
|
73 |
+
model = AutoModelForCausalLM.from_pretrained("DataVortexM-7B-Instruct-v0.1")
|
74 |
+
tokenizer = AutoTokenizer.from_pretrained("DataVortexM-7B-Instruct-v0.1")
|
75 |
|
76 |
messages = [
|
77 |
+
{"role": "system", "content": "λΉμ μ μ¬λλ€μ΄ μ 보λ₯Ό μ°Ύμ μ μλλ‘ λμμ£Όλ μΈκ³΅μ§λ₯ λΉμμ
λλ€."},
|
78 |
+
{"role": "user", "content": "λνλ―Όκ΅μ μλλ μ΄λμΌ?"},
|
79 |
+
{"role": "assistant", "content": "λνλ―Όκ΅μ μλλ μμΈμ
λλ€."},
|
80 |
+
{"role": "user", "content": "μμΈ μΈκ΅¬λ μ΄ λͺ λͺ
μ΄μΌ?"}
|
81 |
]
|
82 |
|
83 |
+
encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
|
84 |
+
|
85 |
+
model_inputs = encodeds.to(device)
|
86 |
+
model.to(device)
|
87 |
+
|
88 |
+
generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
|
89 |
+
decoded = tokenizer.batch_decode(generated_ids)
|
90 |
+
print(decoded[0])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
91 |
```
|
92 |
|
93 |
+
## **License**
|
94 |
+
|
95 |
+
The model is licensed under the [cc-by-nc-sa-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license, which allows others to copy, modify, and share the work non-commercially, as long as they give appropriate credit and distribute any derivative works under the same license.
|
96 |
+
|
97 |
<div align="center">
|
98 |
<a href="https://edentns.com/">
|
99 |
+
<img src="./Logo.png" alt="Logo" style="height: 3em;">
|
100 |
</a>
|
101 |
</div>
|