---
license: cc-by-4.0
library_name: peft
base_model: Sao10K/L3-8B-Stheno-v3.1
language:
- en
pipeline_tag: text2text-generation
---

## Overview

**Secretmoon/LoRA-Llama-3-MLP** is a LoRA adapter for the Llama-3-8B model, primarily designed to expand the model's knowledge of the MLP:FiM universe. This adapter is ideal for generating fan fiction, role-playing scenarios, and other creative projects. The training data includes factual content from the Fandom wiki and canonical fan works that deeply explore the universe.

![Night alicorn](https://huggingface.co/secretmoon/LoRA-Llama-3-MLP/resolve/main/profile.png)

## Base Model

The base model for this adapter is **Sao10K/L3-8B-Stheno-v3.1**, an excellent fine-tuned version of the original Llama-3-8B. It excels in story writing and role-playing without suffering from degradation due to overfitting.

## Training Details

- **Dataset:**
  1. Cleaned copy of the MLP Fandom Wiki, excluding information about recent and side projects unrelated to MLP:FiM. (Alpaca)
  2. Approximately 100 specially selected fan stories from FiMFiction. (RAW text)
  3. Additional data to train the model as a personal assistant and enhance its sensitivity to user emotions. (Alpaca)
- **Training Duration:** 3 hours
- **Hardware:** 1 x NVIDIA RTX A6000 48GB
- **PEFT Type:** LoRA 8-bit
- **Sequence Length:** 6144
- **Batch Size:** 2
- **Num Epochs:** 3
- **Optimizer:** AdamW_BNB_8bit
- **Learning Rate Scheduler:** Cosine
- **Learning Rate:** 0.00033
- **LoRA R:** 256
- **Sample Packing:** True
- **LoRA Target Linear:** True

## How to Use

You can apply the adapter to the original Safetensors weights of the model and load it through Transformers, or you can merge this adapter with the base model weights and convert it to f16 .gguf for use in llama.cpp.

### Recommendations for LoRA Alpha

- **16:** Low influence
- **32:** Suggested optimal value (recommended)
- **64:** High influence, significantly impacting model behavior
- **128:** Very high influence, drastically changing language model behavior (not recommended)

You can modify this parameter in the `adapter_config.json` file. For example, I merged the adapter with the base model using LoRA alpha=40.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
from peft import PeftModel, PeftConfig

# Load the base Llama-3-8B fine-tuned model and tokenizer
base_model_name = "Sao10K/L3-8B-Stheno-v3.1"
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(base_model_name)

# Load the Secretmoon/LoRA-Llama-3-MLP adapter
adapter_name = "secretmoon/LoRA-Llama-3-MLP"
config = PeftConfig.from_pretrained(adapter_name)
adapter_model = PeftModel.from_pretrained(base_model, config)

# Example usage
input_text = "Tell me about the abandoned castle in the Everfree forest."
inputs = tokenizer(input_text, return_tensors="pt")
outputs = adapter_model.generate(**inputs, max_length=300)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```


## Merge:

1. **Using Axolotl** (https://github.com/OpenAccess-AI-Collective/axolotl)
   ```bash
   python3 -m axolotl.cli.merge_lora lora.yml --lora_model_dir="./completed-model"
   ```

2. **Conversion to adapter for gguf in OLD llama.cpp**
  ```bash
   python3 convert-lora-to-ggml.py /path/to/lora/adapter
  ```

## Other:
You can contact me on telegram @monstor86 or discord [PASTE DISCORD]
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)