---
license: mit
datasets:
- isek-ai/danbooru-tags-2016-2023
language:
- en
library_name: transformers
---

# SDPrompt-RetNet-v2-beta

This model is a pretrained RetNet model trained from scratch using https://github.com/syncdoth/RetNet.

It achieves the following results on the evaluation set:
- Loss: 0.5923

## Usage

```bash
pip install transformers safetensors
```

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

MODEL_NAME = "isek-ai/SDPrompt-RetNet-v2-beta"
DEVICE = "cuda"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model= AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.float16, # or torch.bfloat16
    trust_remote_code=True,
).to(DEVICE)
model.eval()
streamer = TextStreamer(tokenizer)

prompt = "1girl"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
_ = model.generate(
    inputs["input_ids"],
    max_new_tokens=256,
    do_sample=True,
    top_p=0.9,
    top_k=20,
    temperature=0.9,
    streamer=streamer,
)
# 1girl, :<, bag, black hair, blurry, bokeh, cloud, depth of field, from side, long sleeves, night, outdoors, pleated skirt, power lines, purple eyes, road, scenery, shoes, shoulder bag,gasm, sidelocks, sign, skirt,let's drawsaurus, skylight smile, sneakers, standing, star (sky), sweater, town, traffic cone, utility pole, vending machine, wide-eyed, window, wooden box, yellow skirt,ization, zettai ryouiki, zoom layer, white footwear, zipper, zipper pull tab, zipperland sheet, zombie pose, ladder, leaning back, leg up, looking to the side,let, miniskirt, motion blur, musical note, open mouth, part
```


## Model description

This model is trained with **only Danbooru tags** to generate prompts for image generation models.

## Training data

- [isek-ai/danbooru-tags-2016-2023](https://huggingface.co/datasets/isek-ai/danbooru-tags-2016-2023)

### Dataset filtering

TODO

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.975         | 0.07  | 500  | 1.0005          |
| 0.7549        | 0.13  | 1000 | 0.7604          |
| 0.6923        | 0.2   | 1500 | 0.7090          |
| 0.6753        | 0.26  | 2000 | 0.6778          |
| 0.6591        | 0.33  | 2500 | 0.6568          |
| 0.6337        | 0.39  | 3000 | 0.6429          |
| 0.6288        | 0.46  | 3500 | 0.6319          |
| 0.624         | 0.53  | 4000 | 0.6218          |
| 0.62          | 0.59  | 4500 | 0.6172          |
| 0.603         | 0.66  | 5000 | 0.6090          |
| 0.5931        | 0.72  | 5500 | 0.6032          |
| 0.5957        | 0.79  | 6000 | 0.5986          |
| 0.5972        | 0.85  | 6500 | 0.5948          |
| 0.5928        | 0.92  | 7000 | 0.5926          |
| 0.5904        | 0.98  | 7500 | 0.5923          |


### Framework versions

- Transformers 4.36.1
- Pytorch 2.1.2+cu121
- Datasets 2.15.0
- Tokenizers 0.15.0