Titovs Aspr commited on
Commit
efc3eaf
1 Parent(s): 45add03

Update README.md (#1)

Browse files

- Update README.md (36660df336790d9bee2a960f4429acdb65b13629)


Co-authored-by: Mikhail Evtikhiev <[email protected]>

Files changed (1) hide show
  1. README.md +93 -3
README.md CHANGED
@@ -1,3 +1,93 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ datasets:
4
+ - JetBrains/KStack
5
+ results:
6
+ - task:
7
+ type: text-generation
8
+ dataset:
9
+ name: MultiPL-HumanEval (Kotlin)
10
+ type: openai_humaneval
11
+ metrics:
12
+ - name: pass@1
13
+ type: pass@1
14
+ value: 29.19
15
+ tags:
16
+ - code
17
+ ---
18
+
19
+ # KStack-full models
20
+
21
+ KStack-full models is a collection of fine-tuned open-source generative text models fine-tuned on KStack dataset.
22
+ This is a repository for fine-tuned CodeLlama-7b model in the Hugging Face Transformers format.
23
+
24
+ # Model use
25
+
26
+ ```python
27
+ from transformers import AutoModelForCausalLM, AutoTokenizer
28
+
29
+ # Load pre-trained model and tokenizer
30
+ model_name = 'JetBrains/CodeLlama-7B-KStack-full'
31
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
32
+ model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')
33
+
34
+ # Create and encode input
35
+ input_text = """\
36
+ This function takes an integer n and returns factorial of a number:
37
+ fun factorial(n: Int): Int {\
38
+ """
39
+ input_ids = tokenizer.encode(
40
+ input_text, return_tensors='pt'
41
+ ).to('cuda')
42
+
43
+ # Generate
44
+ output = model.generate(
45
+ input_ids, max_length=60, num_return_sequences=1,
46
+ pad_token_id=tokenizer.eos_token_id,
47
+ )
48
+
49
+ # Decode output
50
+ generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
51
+ print(generated_text)
52
+ ```
53
+
54
+ As with the base model, we can use FIM. To do this, the following format must be used:
55
+ ```
56
+ '<PRE> ' + prefix + ' <SUF> ' + suffix + ' <MID>'
57
+ ```
58
+
59
+ # Training setup
60
+
61
+ The model was trained on one A100 GPU with following hyperparameters:
62
+
63
+ | **Hyperparameter** | **Value** |
64
+ |:---------------------------:|:----------------------------------------:|
65
+ | `warmup` | 5% |
66
+ | `max_lr` | 1e-6 |
67
+ | `num_epochs` | 1 |
68
+ | 'attention_dropout' | 0.1 |
69
+ | `scheduler` | cosine |
70
+ | `total_batch_size` | 128 (~65K tokens per step) |
71
+ | `num_epochs` | 1 |
72
+
73
+ More details about finetuning can be found in the technical report
74
+
75
+ # Fine-tuning data
76
+
77
+ For this model we used 15K exmaples of Kotlin Exercices dataset {TODO: link!}. Every example follows HumanEval like format. In total dataset contains about 3.5M tokens.
78
+ For more information about the dataset follow the link.
79
+
80
+ # Evaluation
81
+
82
+ To evaluate we used Kotlin Humaneval (more infromation here)
83
+
84
+ Fine-tuned model:
85
+
86
+ | **Model name** | **Kotlin HumanEval Pass Rate** |
87
+ |:---------------------------:|:----------------------------------------:|
88
+ | `base model` | 26.09 |
89
+ | `fine-tuned model` | 29.19 |
90
+
91
+ # Ethical Considerations and Limitations
92
+
93
+ Code Llama and its variants are a new technology that carries risks with use. The testing conducted to date could not cover all scenarios. For these reasons, as with all LLMs, Kexer's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of Kexer, developers should perform safety testing and tuning tailored to their specific applications of the model.