siddartha-abacus commited on
Commit
38ae4df
1 Parent(s): 2830b44

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +24 -1
README.md CHANGED
@@ -2,4 +2,27 @@
2
  license: apache-2.0
3
  ---
4
 
5
- Merge of models, more details to follow.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: apache-2.0
3
  ---
4
 
5
+ Slerp Merge of cookinai/CatMacaroni-Slerp and mncai/mistral-7b-dpo-v5
6
+
7
+ .yaml file for mergekit
8
+
9
+ ```yaml
10
+ slices:
11
+ - sources:
12
+ - model: cookinai/CatMacaroni-Slerp
13
+ layer_range: [0, 32]
14
+ - model: mncai/mistral-7b-dpo-v5
15
+ layer_range: [0, 32]
16
+ merge_method: slerp
17
+ base_model: mncai/mistral-7b-dpo-v5
18
+ parameters:
19
+ t:
20
+ - filter: self_attn
21
+ value: [0, 0.5, 0.3, 0.7, 1]
22
+ - filter: mlp
23
+ value: [1, 0.5, 0.7, 0.3, 0]
24
+ - value: 0.5 # fallback for rest of tensors
25
+ dtype: float16
26
+ ```
27
+
28
+ Models chose to achive a mix of performance on reasoning datasets like GSM8k and conversational tasks.