Model stuck until max_new_tokens?
Hello, I'm using the model for an image captioning task.
I use the prompt:
prompt = f"""[INST]\You are an assistant tasked with summarizing images for image retrieval. \n
Context:\n
{context}[/INST]"""
But sometimes it gets stuck and repeats the same words until it gets to max_new_token. Is there a way to prevent this behavior?
Thanks!
@danielelongo
Does it get repetitive after certain length, maybe the model isn't good at long generations in general. You can mitigate repetitions with repetition_penalty (see https://huggingface.co/docs/transformers/v4.44.2/en/main_classes/text_generation#transformers.GenerationConfig) or you can also verify that the input is formatted correctly, in case the generation quality is bad in general
Thank you very much!