README.md

metadata

license: apache-2.0
language:
  - zh
  - en

Chinese-CodeLlama-7B-SFT-V2

We added 7k+ python code instructions and implemented SFT based on our Chinese-CodeLlama-7B-SFT-V1. Drawing on the work of code-llama, we increased the base period of rotary positional embeddings (RoPE) from 10000 to 1000000.

We use a sequence length of 1k for pre-training, and continue training based on this length during the fine-tuning stage. Based on a larger base period of RoPE, it can support up 15k context length extrapolation at inference time.

Based on this dataset, we calculate the average of PPL on 1k length text to be 5.44. However, this value is 148.70 based on our pre-trained model.

The Chinese prompt template used is as follows:

PROMPT_TEMPLATE = (
  "下面是描述一项任务的指令，并且与一则输入配对用来提供更多的上下文。请给出尽可能满足请求的回答.\n"
  "### 指令:\n{instruction}\n### 输入:\n{input}\n### 回答:\n"
)