Vortex-3b-v2 is an upgraded version of the Vortex-3b model ie. a 2.78 billion parameter causal language model created by OEvortex that was derived from EleutherAI's Pythia-2.8b and trained on 79% of uncensored-vortex dataset
from transformers import pipeline
# Initialize the pipeline
pipe = pipeline("text-generation", model="OEvortex/vortex-3b-v2")
# Use the pipeline
text = "Once upon a time"
generated_text = pipe(text, max_length=100, do_sample=True)[0]['generated_text']
print(generated_text)
# Use a pipeline as a high-level helper
from transformers import pipeline
text = pipeline(model="OEvortex/vortex-3b-v2", torch_dtype=torch.bfloat16, device_map="auto")
res = text("Explain to me the difference between nuclear fission and fusion.")
print(res[0]["text"])
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric | vortex 3b | vortex 3b-v2 | dolly-v2-3b | pythia-2.8b-deduped |
---|---|---|---|---|
Avg. | 35.76 | 37.46 | 25.26 | 36.72 |
AI2 Reasoning Challenge (25-Shot) | 31.91 | 39.68 | 22.83 | 36.26 |
HellaSwag (10-Shot) | 56.89 | 65.04 | 26.55 | 60.66 |
MMLU (5-Shot) | 27.32 | 25.09 | 24.7 | 26.78 |
TruthfulQA (0-shot) | 37.39 | 33.80 | 0 | 35.56 |
Winogrande (5-shot) | 60.14 | 59.12 | 59.43 | 60.22 |
GSM8k (5-shot) | 0.91 | 2.05 | 1.86 | 0.83 |
- Downloads last month
- 83
Inference API (serverless) is not available, repository is disabled.
Dataset used to train OEvortex/vortex-3b-v2
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard39.680
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard65.040
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard25.090
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard33.800
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard59.120
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard2.050