Hugging Face's logo

rhplus0831
/
maid-yuzu-v8-alter

Text Generation
Transformers
Safetensors
mixtral
mergekit
Merge
text-generation-inference
Inference Endpoints
Model card Files Community
2
rhplus0831 commited on
Commit
4fb0512
•
1 Parent(s): 81175c3

Upload folder using huggingface_hub

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +65 -0
  2. config.json +30 -0
  3. mergekit_config.yml +18 -0
  4. model-00001-of-00048.safetensors +3 -0
  5. model-00002-of-00048.safetensors +3 -0
  6. model-00003-of-00048.safetensors +3 -0
  7. model-00004-of-00048.safetensors +3 -0
  8. model-00005-of-00048.safetensors +3 -0
  9. model-00006-of-00048.safetensors +3 -0
  10. model-00007-of-00048.safetensors +3 -0
  11. model-00008-of-00048.safetensors +3 -0
  12. model-00009-of-00048.safetensors +3 -0
  13. model-00010-of-00048.safetensors +3 -0
  14. model-00011-of-00048.safetensors +3 -0
  15. model-00012-of-00048.safetensors +3 -0
  16. model-00013-of-00048.safetensors +3 -0
  17. model-00014-of-00048.safetensors +3 -0
  18. model-00015-of-00048.safetensors +3 -0
  19. model-00016-of-00048.safetensors +3 -0
  20. model-00017-of-00048.safetensors +3 -0
  21. model-00018-of-00048.safetensors +3 -0
  22. model-00019-of-00048.safetensors +3 -0
  23. model-00020-of-00048.safetensors +3 -0
  24. model-00021-of-00048.safetensors +3 -0
  25. model-00022-of-00048.safetensors +3 -0
  26. model-00023-of-00048.safetensors +3 -0
  27. model-00024-of-00048.safetensors +3 -0
  28. model-00025-of-00048.safetensors +3 -0
  29. model-00026-of-00048.safetensors +3 -0
  30. model-00027-of-00048.safetensors +3 -0
  31. model-00028-of-00048.safetensors +3 -0
  32. model-00029-of-00048.safetensors +3 -0
  33. model-00030-of-00048.safetensors +3 -0
  34. model-00031-of-00048.safetensors +3 -0
  35. model-00032-of-00048.safetensors +3 -0
  36. model-00033-of-00048.safetensors +3 -0
  37. model-00034-of-00048.safetensors +3 -0
  38. model-00035-of-00048.safetensors +3 -0
  39. model-00036-of-00048.safetensors +3 -0
  40. model-00037-of-00048.safetensors +3 -0
  41. model-00038-of-00048.safetensors +3 -0
  42. model-00039-of-00048.safetensors +3 -0
  43. model-00040-of-00048.safetensors +3 -0
  44. model-00041-of-00048.safetensors +3 -0
  45. model-00042-of-00048.safetensors +3 -0
  46. model-00043-of-00048.safetensors +3 -0
  47. model-00044-of-00048.safetensors +3 -0
  48. model-00045-of-00048.safetensors +3 -0
  49. model-00046-of-00048.safetensors +3 -0
  50. model-00047-of-00048.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model:
3
+ - mistralai/Mixtral-8x7B-v0.1
4
+ - mistralai/Mixtral-8x7B-Instruct-v0.1
5
+ - jondurbin/bagel-dpo-8x7b-v0.2
6
+ - cognitivecomputations/dolphin-2.7-mixtral-8x7b
7
+ - NeverSleep/Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss
8
+ - ycros/BagelMIsteryTour-v2-8x7B
9
+ - smelborp/MixtralOrochi8x7B
10
+ library_name: transformers
11
+ tags:
12
+ - mergekit
13
+ - merge
14
+
15
+ ---
16
+ # maid-yuzu-v8-alter
17
+
18
+ This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
19
+
20
+ v7's approach worked better than I thought, so I tried something even weirder as a test. I don't think a proper model will come out, but I'm curious about the results.
21
+
22
+ ## Merge Details
23
+ ### Merge Method
24
+
25
+ This model was merged using the SLERP merge method.
26
+
27
+ This models were merged using the SLERP method in the following order:
28
+
29
+ maid-yuzu-v8-base: mistralai/Mixtral-8x7B-v0.1 + mistralai/Mixtral-8x7B-Instruct-v0.1 = 0.5
30
+ maid-yuzu-v8-step1: above + jondurbin/bagel-dpo-8x7b-v0.2 = 0.25
31
+ maid-yuzu-v8-step2: above + cognitivecomputations/dolphin-2.7-mixtral-8x7b = 0.25
32
+ maid-yuzu-v8-step3: above + NeverSleep/Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss = 0.25
33
+ maid-yuzu-v8-step4-alter: above + ycros/BagelMIsteryTour-v2-8x7B = 0.5
34
+ maid-yuzu-v8-alter: above + smelborp/MixtralOrochi8x7B = 0.5
35
+
36
+ ### Models Merged
37
+
38
+ The following models were included in the merge:
39
+ * [smelborp/MixtralOrochi8x7B](https://huggingface.co/smelborp/MixtralOrochi8x7B)
40
+ * ../maid-yuzu-v8-step4-alter
41
+
42
+ ### Configuration
43
+
44
+ The following YAML configuration was used to produce this model:
45
+
46
+ ```yaml
47
+ base_model:
48
+ model:
49
+ path: ../maid-yuzu-v8-step4-alter
50
+ dtype: bfloat16
51
+ merge_method: slerp
52
+ parameters:
53
+ t:
54
+ - value: 0.5
55
+ slices:
56
+ - sources:
57
+ - layer_range: [0, 32]
58
+ model:
59
+ model:
60
+ path: ../maid-yuzu-v8-step4-alter
61
+ - layer_range: [0, 32]
62
+ model:
63
+ model:
64
+ path: smelborp/MixtralOrochi8x7B
65
+ ```
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "../maid-yuzu-v8-step4-alter",
3
+ "architectures": [
4
+ "MixtralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 4096,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 14336,
13
+ "max_position_embeddings": 32768,
14
+ "model_type": "mixtral",
15
+ "num_attention_heads": 32,
16
+ "num_experts_per_tok": 2,
17
+ "num_hidden_layers": 32,
18
+ "num_key_value_heads": 8,
19
+ "num_local_experts": 8,
20
+ "output_router_logits": false,
21
+ "rms_norm_eps": 1e-05,
22
+ "rope_theta": 1000000.0,
23
+ "router_aux_loss_coef": 0.02,
24
+ "sliding_window": null,
25
+ "tie_word_embeddings": false,
26
+ "torch_dtype": "bfloat16",
27
+ "transformers_version": "4.37.2",
28
+ "use_cache": true,
29
+ "vocab_size": 32000
30
+ }
mergekit_config.yml ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ base_model:
2
+ model:
3
+ path: ../maid-yuzu-v8-step4-alter
4
+ dtype: bfloat16
5
+ merge_method: slerp
6
+ parameters:
7
+ t:
8
+ - value: 0.5
9
+ slices:
10
+ - sources:
11
+ - layer_range: [0, 32]
12
+ model:
13
+ model:
14
+ path: ../maid-yuzu-v8-step4-alter
15
+ - layer_range: [0, 32]
16
+ model:
17
+ model:
18
+ path: smelborp/MixtralOrochi8x7B
model-00001-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cee2097cd6ebe8a6ff1801bd14e9c26079847321fa91bf0d0b367814ccd235ea
3
+ size 1963199472
model-00002-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7f5d7e84ae98752fe2a26042eee7c9a92e87e98fc5b17ba56ba3935fab210f5d
3
+ size 1996490952
model-00003-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:007683201cb1bb7d548c13efb22ba1fba36a55b87ef62cf6502a637d8b0d81c3
3
+ size 1956793496
model-00004-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9edfe15f2cf409c382e939d3154290c9084e0bd96d54e94733fc6205eb5905d3
3
+ size 1996490952
model-00005-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0f5bd46e60f55130733bbf266cd127f64bcd427c76b87e8d8701e4cf0dc887c5
3
+ size 1996590288
model-00006-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ccca957a396640850968b1e54377d9a64937dca1e7338b56007e5dfca809ff5a
3
+ size 1929447960
model-00007-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ad58845833964d1ebd7a618ca8e221fdefa28bf28f4bca695ced5f96e39b4b64
3
+ size 1962953472
model-00008-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07f85ba846d4f2076e6293fc3012c63d0a4cdda9238bc53026684692a11ec295
3
+ size 1996573208
model-00009-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac5b4369acb162fc24d3caef8144796292f59f6470e582457f186d2efc548450
3
+ size 1962936856
model-00010-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:317e8b1c92d23488baccf37e543b2aa1481eb3656244bdef2f7c145dfff644b2
3
+ size 1996507568
model-00011-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:27febc6657dfda8fa7e80447fbc113d917ad4d0c4053ab8e48dff621a99c3cd6
3
+ size 1963068152
model-00012-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:18aac0edb50764184766ab2436f4247b0364469994be8a35a6c4d362d2e8bc03
3
+ size 1996490952
model-00013-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9ab2a86e10b0b407a6bc93640a7662038de5594b6b9b29ac2b32da5f52c832bd
3
+ size 1929547296
model-00014-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:63b8d983915e618f3a289dfdbfb16c800a0ba8ffab943f336b504a0df7ce4aa9
3
+ size 1996490952
model-00015-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7feee1be0f8ee9df33ff78ceb8b50a663926ba9ba0b11b6389964147b4ec2392
3
+ size 1962953472
model-00016-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c080047f7048167c262042cd83c8b186f03720d50bd384bde57d41e3b9265440
3
+ size 1996573240
model-00017-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a05e3f868e8bf08c721d22e92af5c63d07eab7a687d29c8217944d09257f0b01
3
+ size 1962936872
model-00018-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84432f694d6afa63faa69b4c42f325b95a588861053e64301177c2e24f8df230
3
+ size 1996573240
model-00019-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:317e65a6080356e3e927a46365345e5d4886c255371ea0e3ff767b871d8ca014
3
+ size 1963002528
model-00020-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1011e13a6dd83985fa684de941df02ae0fa9da4f7c5d4847215ea2d73eed256e
3
+ size 1962953488
model-00021-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15cd30232062181829595928c4959dff2afcf50c98b3dc7681153351962a9b21
3
+ size 1963084800
model-00022-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f242f4ed6ea97771c6d1d77513cc07a5cdfe4eab269f488dee51ea81d74d8a8b
3
+ size 1962953488
model-00023-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9974491dd9dac342c44e1d64d96adf359e2be94e84bcec7f8fb00f9e8b986d3c
3
+ size 1996573240
model-00024-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe79abab074582c28d8453da17887557a194cfbe7fb07f68f6ad4b82a8f4f380
3
+ size 1962936872
model-00025-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:43a8d30fa7d78465f10b1183b79e009b2170ba86ee3c81be4a0b632faed9a805
3
+ size 1996490968
model-00026-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4abb30a84a00a67787a94c3bc4e9e5ad9ab60053fa4f97a293423237b94da7ab
3
+ size 1996573232
model-00027-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:82b3a0a9bd07550aef349d788ad57e06431c2595bd89fa496e5cf38b38f9f864
3
+ size 1963002528
model-00028-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:862fd8b14ec9012d8bb7d45ced643e9e666cac3b749105fa85d80f2ce830559a
3
+ size 1963035760
model-00029-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f9df935dc4d0a5f2b26daf2f08514b3469b3ac0f1f6c0979ac6f57fedf38d54
3
+ size 1963002528
model-00030-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:94cfa1af69769a436f060f99d8e3687ef1ac7fcd0849f87f28d92588fb575456
3
+ size 1962953488
model-00031-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b283bb853e469a1be488b21efdfb81d1eee5788dd9e57a52b202accf926fb20
3
+ size 1963084800
model-00032-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:deffb4591e678db10cb778f0ce16bb3e50e67701cd7a8c45e9638fd038b575f5
3
+ size 1962953488
model-00033-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e4308e5647b8fc7f05785238a0e3105be4b87c970b4dc7ce75e56da1ad229c90
3
+ size 1996573240
model-00034-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b5574f03ae5808786424df0d8ea7832f72c08128816409b9f8d0f3c054d80596
3
+ size 1962936872
model-00035-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b006ed53653f8d9cbd3fee07e72ed28850fd0153ffd4d2a901ad3e93d6b80b30
3
+ size 1996490968
model-00036-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4bf91d63b65ab6ba7f6e13cb62c7e7029db2d62e80200054a0a93b2acd9812c5
3
+ size 1963084800
model-00037-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:48266d8576197200c12d2a7b8ea0ffc3ce1e5b486ef3e58be03c29701e2c718f
3
+ size 1996490968
model-00038-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:01e866ae07ea870d8ce470532fce5e6c64b4f1824cf8d5d1c652d84f7b0252fb
3
+ size 1929547328
model-00039-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e02cf699848149c3db19072fe9a3ba451588c2f135d78c7d82925efcf7a77c17
3
+ size 1996490968
model-00040-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3dffe94ab305d82dcda8108b7c21fe2664f4a20474559ce3a2ec697e994142fd
3
+ size 1962953488
model-00041-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9eb1026ed48a7c74eea25d722c68834db23809adff588d6b3a51d55ebba05d72
3
+ size 1996507576
model-00042-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c1bbbdb3d720baf3f644ed69e10bee2cefb85717bdbf9afc3724ea2d3fe86191
3
+ size 1962936872
model-00043-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f1d2049810360efb30cc60bae1080a6f3657687c6561bb4f792e12d316147c8a
3
+ size 1996507584
model-00044-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:757c9ea500186a456f44ca48c9f64d31238f0f96ca869748cba16464ef204cec
3
+ size 1963002528
model-00045-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eb016d8154ead4e0aac4871b45a6e2a97212a6a31f11064e3b65e4408c5aa864
3
+ size 1996490968
model-00046-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:17609baed236f246442f230078900f68986dbb1502ed8e84e823cc49e5f9d22f
3
+ size 1990306840
model-00047-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c07b953b76cf30a7c8409969fe5d148ee699a9ba5534a2e07e3cae8d37cdebbc
3
+ size 1962953488