|
--- |
|
language: |
|
- zh |
|
- en |
|
tags: |
|
- glm |
|
- visualglm |
|
- chatglm |
|
- thudm |
|
--- |
|
# VisualGLM-6B |
|
<p align="center"> |
|
💻 <a href="https://github.com/THUDM/VisualGLM-6B" target="_blank">Github Repo</a> • 🐦 <a href="https://twitter.com/thukeg" target="_blank">Twitter</a> • 📃 <a href="https://arxiv.org/abs/2103.10360" target="_blank">[GLM@ACL 22]</a> <a href="https://github.com/THUDM/GLM" target="_blank">[GitHub]</a> • 📃 <a href="https://arxiv.org/abs/2210.02414" target="_blank">[GLM-130B@ICLR 23]</a> <a href="https://github.com/THUDM/GLM-130B" target="_blank">[GitHub]</a> <br> |
|
</p> |
|
|
|
<p align="center"> |
|
👋 Join our <a href="https://join.slack.com/t/chatglm/shared_invite/zt-1th2q5u69-7tURzFuOPanmuHy9hsZnKA" target="_blank">Slack</a> and <a href="https://github.com/THUDM/ChatGLM-6B/blob/main/resources/WECHAT.md" target="_blank">WeChat</a> |
|
</p> |
|
|
|
## 介绍 |
|
VisualGLM-6B 是一个开源的，支持**图像、中文和英文**的多模态对话语言模型，语言模型基于 [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B)，具有 62 亿参数；图像部分通过训练 [BLIP2-Qformer](https://arxiv.org/abs/2301.12597) 构建起视觉模型与语言模型的桥梁，整体模型共78亿参数。 |
|
|
|
VisualGLM-6B 依靠来自于 [CogView](https://arxiv.org/abs/2105.13290) 数据集的30M高质量中文图文对，与300M经过筛选的英文图文对进行预训练，中英文权重相同。该训练方式较好地将视觉信息对齐到ChatGLM的语义空间；之后的微调阶段，模型在长视觉问答数据上训练，以生成符合人类偏好的答案。 |
|
|
|
## 软件依赖 |
|
|
|
```shell |
|
pip install SwissArmyTransformer>=0.3.6 torch>1.10.0 torchvision transformers>=4.27.1 cpm_kernels |
|
``` |
|
|
|
## 代码调用 |
|
|
|
可以通过如下代码调用 VisualGLM-6B 模型来生成对话： |
|
|
|
```ipython |
|
>>> from transformers import AutoTokenizer, AutoModel |
|
>>> tokenizer = AutoTokenizer.from_pretrained("THUDM/visualglm-6b", trust_remote_code=True) |
|
>>> model = AutoModel.from_pretrained("THUDM/visualglm-6b", trust_remote_code=True).half().cuda() |
|
>>> image_path = "your image path" |
|
>>> response, history = model.chat(tokenizer, image_path, "描述这张图片。", history=[]) |
|
>>> print(response) |
|
>>> response, history = model.chat(tokenizer, image_path, "这张图片可能是在什么场所拍摄的？", history=history) |
|
>>> print(response) |
|
``` |
|
|
|
关于更多的使用说明，包括如何运行命令行和网页版本的 DEMO，以及使用模型量化以节省显存，请参考我们的 [Github Repo](https://github.com/THUDM/VisualGLM-6B)。 |
|
|
|
For more instructions, including how to run CLI and web demos, and model quantization, please refer to our [Github Repo](https://github.com/THUDM/VisualGLM-6B). |
|
|
|
## 协议 |
|
|
|
本仓库的代码依照 [Apache-2.0](LICENSE) 协议开源，VisualGLM-6B 模型的权重的使用则需要遵循 [Model License](MODEL_LICENSE)。 |
|
|
|
## 引用 |
|
|
|
如果你觉得我们的工作有帮助的话，请考虑引用下列论文： |
|
|
|
``` |
|
@inproceedings{du2022glm, |
|
title={GLM: General Language Model Pretraining with Autoregressive Blank Infilling}, |
|
author={Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie}, |
|
booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, |
|
pages={320--335}, |
|
year={2022} |
|
} |
|
``` |
|
``` |
|
@article{ding2021cogview, |
|
title={Cogview: Mastering text-to-image generation via transformers}, |
|
author={Ding, Ming and Yang, Zhuoyi and Hong, Wenyi and Zheng, Wendi and Zhou, Chang and Yin, Da and Lin, Junyang and Zou, Xu and Shao, Zhou and Yang, Hongxia and others}, |
|
journal={Advances in Neural Information Processing Systems}, |
|
volume={34}, |
|
pages={19822--19835}, |
|
year={2021} |
|
} |
|
``` |
