Failed model (anthracite-org/magnum-v2.5-12b-kto)

#905
by CombinHorizon - opened

request file: https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/anthracite-org/magnum-v2.5-12b-kto_eval_request_False_float16_Original.json

webpage: anthracite-org/magnum-v2.5-12b-kto (float16)

since it uses KTO & DPO-P and has a generation_config.json, shouldn't it rather be re-classified as a chat-model?

any info on why it failed (logs), i guess it looks like this is a request for a re-run, (esp if it is due to external factors)

Open LLM Leaderboard org

Hi @CombinHorizon ,

According to the status in the provided request file, the model is in PENDING and will be soon evaluated. I should also note, as stated in our FAQ, please, do not resubmit failed models via "Submit" tab, but instead feel free to open a discussion to ask for the resubmission / changing the request file.

I suppose yes, this model can be classified as a chat model.

I close this issue, please, ping me here in case of any other questions about this model or feel free to open a new discussion!

alozowski changed discussion status to closed

Sign up or log in to comment