Skip to content

[Bug]: vllm eval failed #2414

Description

@wenhuach21

Problem Description

auto-round qwen3-8b-quantized --eval --eval_backend vllm --tasks gsm8k
2026-09-24 13:58:37 INFO eval_cli.py L324: Detected existing CUDA_VISIBLE_DEVICES=3, skipping overwrite; tensor_parallel_size=1 derived from CUDA_VISIBLE_DEVICES
The device argument is ignored by the vLLM backend. To select which GPU(s) vLLM can use, set the CUDA_VISIBLE_DEVICES environment variable before running lm-eval.
INFO 09-24 13:58:37 [api_utils.py:286] non-default args: {'trust_remote_code': True, 'seed': 1234, 'disable_log_stats': True, 'allow_deprecated_quantization': True, 'model': './'}
INFO 09-24 13:58:50 [model.py:692] Resolved architecture: Qwen3ForCausalLM
INFO 09-24 13:58:50 [model.py:2030] Using max model len 40960
INFO 09-24 13:58:50 [scheduler.py:288] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 09-24 13:58:50 [kernel.py:416] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=2167076) INFO 09-24 13:58:53 [core.py:123] Initializing a V1 LLM engine (v0.30.0) with config: model='./', speculative_config=None, tokenizer='./', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=1234, served_model_name=./, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] EngineCore failed to start.
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] Traceback (most recent call last):
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/v1/engine/core.py", line 1328, in run_engine_core
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] return func(*args, **kwargs)
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/v1/engine/core.py", line 1085, in init
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] super().init(
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ~~~~~~~~~~~~~~~~^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] vllm_config,
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ^^^^^^^^^^^^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ...<3 lines>...
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] internal_dp_balancing,
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] )
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/v1/engine/core.py", line 134, in init
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] self.model_executor = executor_class(vllm_config)
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ~~~~~~~~~~~~~~^^^^^^^^^^^^^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] return func(*args, **kwargs)
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/v1/executor/abstract.py", line 110, in init
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] self._init_executor()
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ~~~~~~~~~~~~~~~~~~~^^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/v1/executor/uniproc_executor.py", line 69, in _init_executor
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] self.driver_worker.init_device()
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/v1/worker/worker_base.py", line 355, in init_device
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] self.worker.init_device() # type: ignore
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ~~~~~~~~~~~~~~~~~~~~~~~^^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] return func(*args, **kwargs)
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/vllm/v1/worker/gpu_worker.py", line 424, in init_device
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] torch.accelerator.set_device_index(self.device)
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/torch/accelerator/init.py", line 198, in set_device_index
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] torch._C._accelerator_setDeviceIndex(device_index)
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/torch/cuda/init.py", line 517, in _lazy_init
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] raise RuntimeError(
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] ...<2 lines>...
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] )
(EngineCore pid=2167076) ERROR 09-24 13:58:56 [core.py:1366] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

Reproduction Steps

~

Environment Information

No response

Error Logs

Additional Context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions