Qwen3.5-9B-INT4

Repository
RedHatAI/Qwen3.5-9B-quantized.w4a16
Engine
vLLM
Listed
16 Nov 2026

This model is a quantized version of Qwen/Qwen3.5-9B. Evaluation results and reproduction steps are provided below.

Serves
Chat, Reasoning, Multimodal input
Parameters
8.2B
Context
262,144 tokens
Weights
11.5 GB
License
Apache-2.0

Capabilities

Read from the model's capability tags. Requests are admitted per tag.

Chat chat
  • POST /v1/chat/completions
  • POST /v1/responses

Endpoint task mode generate

Inputs vision
Text, Images
Multimodal input: request parts beyond text are admitted per tag.
Reasoning effort reasoning_effort
none, low, medium, high
Values a request may send; the model family sets the vocabulary.
Structured output structured_output
supported
Proven by probe: a JSON schema request returned a payload that parses.
Forced tool choice forced_tool_choice
supported
A forced tool_choice returns a call to that tool.

Launch settings

Every engine serving the model starts with these.

Set by the owner. auto: the serving node derives it. default: the platform's value, left in place.

Engine
vLLM
Context window --max-model-len
262,144 tokens
Quantization --quantization
none default
Weights ship quantized (compressed-tensors, 4-bit).
KV cache dtype --kv-cache-dtype
auto
No flag: the checkpoint's KV quantization, else the model dtype.
Scheduling policy --scheduling-policy
priority default
Interactive requests ahead of batch work; same-class traffic in arrival order.
Tool-call parser --tool-call-parser
auto
Resolves to qwen3_coder from the architecture.
Reasoning parser --reasoning-parser
auto
Resolves to qwen3 from the architecture.
Trust remote code --trust-remote-code
on default
Python shipped in the bundle may run at load (the default).

Engine tuning

Every setting auto: the serving node derives each one.

Set by the owner. auto: the serving node derives it.

Mixture of experts

Expert parallel --enable-expert-parallel
auto
Expert load balancing --enable-eplb
auto
All-to-all backend --all2all-backend
auto

Memory and batching

GPU memory utilization --gpu-memory-utilization
auto
Max sequences --max-num-seqs
auto
Max batched tokens --max-num-batched-tokens
auto
Long-prefill threshold --long-prefill-token-threshold
auto

Execution

Enforce eager --enforce-eager
auto
Prefix caching --enable-prefix-caching
auto
Async scheduling --no-async-scheduling
auto
Custom all-reduce --disable-custom-all-reduce
auto

Speculative decoding

Draft tokens checked by the model in one pass.

Speculative decoding
not configured
An endpoint's own setting, or the serving node's default for the model family, applies.

Architecture

Read from the model's files at upload.

Architecture
qwen3_5
Parameters
8,208,515,072
Layers num_hidden_layers
32
Hidden size hidden_size
4,096
Attention heads num_attention_heads
16 (4 KV)
Head dim head_dim
256
Vocabulary vocab_size
248,320
Native dtype torch_dtype
bfloat16
Weights on disk
11.5 GB, safetensors

Call it

Create an endpoint for the model, then send it a request in the OpenAI shape.

Shell
# Create an endpoint for this model, then provision it. # --tier-id is the service tier to run on; pick one at /endpoints/new. erectl endpoints create \ --name "My endpoint" \ --slug my-endpoint \ --model-id 5CA770EA-D9E6-4DC3-AB25-3B2B6E092D61 \ --tier-id <tier-id> # provision takes the endpoint UUID returned by create erectl endpoints provision <endpoint-id> # Call it. Requests are routed by /<project-id>/<endpoint-slug>. curl https://api.erebine.ai/proj_ABC123/my-endpoint/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $EREBINE_API_KEY" \ -d '{ "model": "Qwen3.5-9B-INT4", "messages": [{"role": "user", "content": "Hello, world!"}], "stream": true }'

Model card

As published with RedHatAI/Qwen3.5-9B-quantized.w4a16. Images are left out; its HTML tables are set as tables.

Model Overview

  • Model Architecture: Qwen/Qwen3.5-9B
  • Input: Text / Image
  • Output: Text
  • Model Optimizations:
  • Weight quantization: INT4
  • Activation quantization: None
  • Model size: 11 GB (reduced from 19.3 GB in BF16)
  • Release Date: 2026-04-24
  • Version: 1.0
  • Model Developers: RedHatAI

Model Optimizations

This model was obtained by quantizing the weights of Qwen/Qwen3.5-9B to INT4 data type while keeping activations in original precision, ready for inference with vLLM.

This optimization reduces the model weights from 19.3 GB to 11 GB on disk (~43% reduction). The reduction is less than the theoretical 75% because the vision encoder, token embeddings, and linear attention layers remain in BF16.

Only the weights of the linear operators within transformer blocks are quantized using LLM Compressor. The vision encoder, token embeddings, and linear attention layers are not quantized.

Deployment

Use with vLLM

  1. Initialize vLLM server:

Multimodal (vision + text):

vllm serve RedHatAI/Qwen3.5-9B-quantized.w4a16 \
  --reasoning-parser qwen3 \
  --max-model-len 262144

Text-only (lower memory):

vllm serve RedHatAI/Qwen3.5-9B-quantized.w4a16 \
  --reasoning-parser qwen3 \
  --max-model-len 262144 \
  --language-model-only
  1. Send requests to the server:
from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

model = "RedHatAI/Qwen3.5-9B-quantized.w4a16"

messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
    model=model,
    messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

This model was created by applying LLM Compressor with calibration samples from Open-Platypus, as presented in the code snippet below.

Details
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from datasets import load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from transformers import AutoProcessor, AutoTokenizer, Qwen3_5ForConditionalGeneration

MODEL_ID = "Qwen/Qwen3.5-9B"
NUM_CALIBRATION_SAMPLES = 1024
MAX_SEQUENCE_LENGTH = 8192

IGNORE_LAYERS = [
    "re:.*lm_head",
    "re:.*embed_tokens$",
    "re:.*visual.*",
    "re:.*model.visual.*",
    "re:.*linear_attn.*",
]

model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)

ds = load_dataset("garage-bAInd/Open-Platypus", split=f"train[:{NUM_CALIBRATION_SAMPLES}]")
ds = ds.shuffle(seed=42)

def preprocess(ex):
    text = ex["instruction"]
    if ex.get("input"):
        text += "\n" + ex["input"]
    return {"text": text}

def tokenize(sample):
    return tokenizer(
        sample["text"],
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )

ds = ds.map(preprocess).map(tokenize, remove_columns=ds.column_names)

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    sequential_targets=["Qwen3_5DecoderLayer"],
    ignore=IGNORE_LAYERS,
    dampening_frac=0.05,
)

oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

model.save_pretrained("Qwen3.5-9B-quantized.w4a16", save_compressed=True)
processor.save_pretrained("Qwen3.5-9B-quantized.w4a16")
save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir="Qwen3.5-9B-quantized.w4a16")
Package versions
  • llm-compressor==0.10.1.dev44+g437f8afe
  • compressed-tensors==0.14.1a20260325
  • transformers==5.3.0
  • vllm==0.18.1
  • lm-eval — neuralmagic/lm-evaluation-harness@741f1d8 (branch: mmlu-pro-chat-variant)
  • lighteval — neuralmagic/lighteval@6f0f351 (branch: eldar-fix-litellm)

Evaluation

This model was evaluated on GSM8k-Platinum, MMLU-Pro, IFEval, Math 500, AIME 2025, and GPQA Diamond using lm-evaluation-harness and lighteval, with inference served via vLLM.

Accuracy

A table that spans rows is left out; see the repository.

Reproduction

The results were obtained using the following commands. GSM8k-Platinum, MMLU-Pro, IFEval, Math 500, and GPQA Diamond were each run 3 times with different seeds and results averaged. AIME 2025 was run 8 times. The vLLM server was started with --language-model-only for all evaluations.

Details
GSM8k-Platinum (lm-eval, 0-shot, 3 repetitions)
lm_eval --model local-chat-completions \
  --tasks gsm8k_platinum_cot_llama \
  --model_args "model=inference-optimization/Qwen3.5-9B-quantized.w4a16,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_gsm8k_platinum.json \
  --seed  \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed="
Seeds used: 42, 1234, 4158
MMLU-Pro (lm-eval, 0-shot, 3 repetitions)
lm_eval --model local-chat-completions \
  --tasks mmlu_pro_chat \
  --model_args "model=inference-optimization/Qwen3.5-9B-quantized.w4a16,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_mmlu_pro.json \
  --seed  \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed="
Seeds used: 42, 1234, 4158
IFEval (lm-eval, 0-shot, 3 repetitions)
lm_eval --model local-chat-completions \
  --tasks ifeval \
  --model_args "model=RedHatAI/Qwen3.5-9B-quantized.w4a16,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_ifeval.json \
  --seed  \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed="
Seeds used: 42, 1234, 4158
Math 500 (lighteval, 0-shot, 3 repetitions)
lighteval endpoint litellm \
  "model_name=hosted_vllm/RedHatAI/Qwen3.5-9B-quantized.w4a16,provider=hosted_vllm,base_url=http://0.0.0.0:8000/v1,timeout=3600,concurrent_requests=100,generation_parameters={temperature:1.0,max_new_tokens:65536,top_p:0.95,top_k:20,min_p:0.0,presence_penalty:1.5,repetition_penalty:1.0,seed:}" \
  "math_500@k=1@n=1|0" \
  --output-dir results_math500 \
  --save-details
Seeds used: 42, 1234, 4158
AIME 2025 (lighteval, 0-shot, 8 repetitions)
lighteval endpoint litellm \
  "model_name=hosted_vllm/RedHatAI/Qwen3.5-9B-quantized.w4a16,provider=hosted_vllm,base_url=http://0.0.0.0:8000/v1,timeout=3600,concurrent_requests=100,generation_parameters={temperature:1.0,max_new_tokens:65536,top_p:0.95,top_k:20,min_p:0.0,presence_penalty:1.5,repetition_penalty:1.0,seed:}" \
  "aime25@k=1@n=1|0" \
  --output-dir results_aime25 \
  --save-details
Seeds used: 42, 1234, 1356, 3344, 4158, 5322, 5678, 9843
GPQA Diamond (lighteval, 0-shot, 3 repetitions)
lighteval endpoint litellm \
  "model_name=hosted_vllm/RedHatAI/Qwen3.5-9B-quantized.w4a16,provider=hosted_vllm,base_url=http://0.0.0.0:8000/v1,timeout=3600,concurrent_requests=100,generation_parameters={temperature:1.0,max_new_tokens:65536,top_p:0.95,top_k:20,min_p:0.0,presence_penalty:1.5,repetition_penalty:1.0,seed:}" \
  "gpqa:diamond@k=1@n=1|0" \
  --output-dir results_gpqa_diamond \
  --save-details
Seeds used: 42, 1234, 4158