patrickvonplaten's picture
Update README.md
7433705
metadata
license: apache-2.0
datasets:
  - laion/laion-art
language:
  - en
library_name: diffusers
pipeline_tag: image-to-image
tags:
  - jax-diffusers-event
base_model: runwayml/stable-diffusion-v1-5

Color-Canny CantrolNet

These are ControlNet checkpoints trained on runwayml/stable-diffusion-v1-5, using fused color and canny edge as conditioning.

You can find some example images in the following.

Examples

Color examples

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

negative prompt: text, bad anatomy, blurry, (low quality, blurry) images_1)

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

negative prompt: text, bad anatomy, blurry, (low quality, blurry) images_2)

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the grass

negative prompt: text, bad anatomy, blurry, (low quality, blurry) images_3)

Brightness examples

This model also can be used to control image brightness. The following images are generated with different brightness conditioning image and controlnet strength(0.5 ~ 0.7). images_4)

Limitations and Bias

  • No strict control by input color
  • Sometimes generate image with confusion When color description in prompt

Training

Dataset We train this model on laion-art dataset with 2.6m images, the processed dataset can be found in ghoskno/laion-art-en-colorcanny.

Training Details

  • Hardware: Google Cloud TPUv4-8 VM

  • Optimizer: AdamW

  • Train Batch Size: 4 x 4 = 16

  • Learning rate: 0.00001 constant

  • Gradient Accumulation Steps: 4

  • Resolution: 512

  • Train Steps: 36000