vllm: ....does not appear to have a file named config.json
#1
by
paolovic
- opened
Hi,
since vLLM still requires GGUF models to have a config.json, is there the possibility to create one?
Thank you in advance
Best regards
Hi
@bartowski
,
thanks for your quick response!
With the config.json I get the following error:
File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/runtime_resources/pip/abc/venv/lib64/python3.11/site-packages/ray/serve/_private/replica.py", line 631, in initialize_and_get_metadata
raise RuntimeError(traceback.format_exc()) from None
RuntimeError: Traceback (most recent call last):
File "/runtime_resources/pip/abc/venv/lib64/python3.11/site-packages/ray/serve/_private/replica.py", line 609, in initialize_and_get_metadata
await self._user_callable_wrapper.initialize_callable()
File "/runtime_resources/pip/abc/venv/lib64/python3.11/site-packages/ray/serve/_private/replica.py", line 901, in initialize_callable
await self._call_func_or_gen(
File "/runtime_resources/pip/abc/venv/lib64/python3.11/site-packages/ray/serve/_private/replica.py", line 867, in _call_func_or_gen
result = callable(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/runtime_resources/pip/abc/venv/lib64/python3.11/site-packages/ray/serve/api.py", line 219, in __init__
cls.__init__(self, *args, **kwargs)
File "/u01/app/mlo/projects/llm-apis/ray_vllm_inference/vllm_serve.py", line 105, in __init__
self.engine = AsyncLLMEngine.from_engine_args(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/runtime_resources/pip/abc/venv/lib64/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 726, in from_engine_args
engine_config = engine_args.create_engine_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/runtime_resources/pip/abc/venv/lib64/python3.11/site-packages/vllm/engine/arg_utils.py", line 792, in create_engine_config
model_config = ModelConfig(
^^^^^^^^^^^^
File "/runtime_resources/pip/abc/venv/lib64/python3.11/site-packages/vllm/config.py", line 230, in __init__
self.multimodal_config = self._init_multimodal_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/runtime_resources/pip/abc/venv/lib64/python3.11/site-packages/vllm/config.py", line 245, in _init_multimodal_config
if any(
^
TypeError: 'NoneType' object is not iterable
But, I'll try to link the model and tokenizer to vllm like they explained here
https://github.com/vllm-project/vllm/issues/8401