gemma-4-31B-it-DFlash
- Engine
- vLLM
- Listed
- 16 Oct 2026
vLLM: until Gemma4 DFlash support is merged, install vLLM from PR #41703:
- Serves
- Chat, Reasoning
- Parameters
- 4.0B
- Context
- 262,144 tokens
- Weights
- 3.1 GB
- License
- Apache-2.0
Capabilities
Read from the model's capability tags. Requests are admitted per tag.
- Chat
chat -
POST /v1/chat/completionsPOST /v1/responses
Endpoint task mode
generate
- Inputs
- Text
- Reasoning effort
reasoning_effort - none, low, medium, high
- Values a request may send; the model family sets the vocabulary.
- Structured output
structured_output - not yet probed
- Admitted until the capability probe observes the model.
- Forced tool choice
forced_tool_choice - not yet probed
- Admitted until the capability probe observes the model.
Launch settings
Every engine serving the model starts with these.
Set by the owner. auto: the serving node derives it. default: the platform's value, left in place.
- Engine
- vLLM
- Context window
--max-model-len - 262,144 tokens
- Quantization
--quantization - none default
- Weights load as stored, bfloat16.
- KV cache dtype
--kv-cache-dtype - auto
- No flag: the checkpoint's KV quantization, else the model dtype.
- Scheduling policy
--scheduling-policy - priority default
- Interactive requests ahead of batch work; same-class traffic in arrival order.
- Tool-call parser
--tool-call-parser - auto
- Resolves to qwen3_coder from the architecture.
- Reasoning parser
--reasoning-parser - auto
- Resolves to qwen3 from the architecture.
- Trust remote code
--trust-remote-code - on default
- Python shipped in the bundle may run at load (the default).
Engine tuning
Every setting auto: the serving node derives each one.
Set by the owner. auto: the serving node derives it.
Mixture of experts
- Expert parallel
--enable-expert-parallel - auto
- Expert load balancing
--enable-eplb - auto
- All-to-all backend
--all2all-backend - auto
Memory and batching
- GPU memory utilization
--gpu-memory-utilization - auto
- Max sequences
--max-num-seqs - auto
- Max batched tokens
--max-num-batched-tokens - auto
- Long-prefill threshold
--long-prefill-token-threshold - auto
Execution
- Enforce eager
--enforce-eager - auto
- Prefix caching
--enable-prefix-caching - auto
- Async scheduling
--no-async-scheduling - auto
- Custom all-reduce
--disable-custom-all-reduce - auto
Speculative decoding
Draft tokens checked by the model in one pass.
- Speculative decoding
- not configured
- An endpoint's own setting, or the serving node's default for the model family, applies.
Architecture
Read from the model's files at upload.
- Architecture
- qwen3
- Parameters
- 4,029,731,328
- Layers
num_hidden_layers - 5
- Hidden size
hidden_size - 5,376
- Attention heads
num_attention_heads - 64 (8 KV)
- Head dim
head_dim - 128
- Vocabulary
vocab_size - 262,144
- Native dtype
torch_dtype - bfloat16
- Weights on disk
- 3.1 GB, safetensors
Call it
Create an endpoint for the model, then send it a request in the OpenAI shape.
# Create an endpoint for this model, then provision it.
# --tier-id is the service tier to run on; pick one at /endpoints/new.
erectl endpoints create \
--name "My endpoint" \
--slug my-endpoint \
--model-id 3AEB3B9F-0B02-42F6-B4C8-6DB551945051 \
--tier-id <tier-id>
# provision takes the endpoint UUID returned by create
erectl endpoints provision <endpoint-id>
# Call it. Requests are routed by /<project-id>/<endpoint-slug>.
curl https://api.erebine.ai/proj_ABC123/my-endpoint/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $EREBINE_API_KEY" \
-d '{
"model": "gemma-4-31B-it-DFlash",
"messages": [{"role": "user", "content": "Hello, world!"}],
"stream": true
}'Model card
Images are left out; its HTML tables are set as tables.
DFlash is a speculative decoding method that uses a lightweight block diffusion model to draft multiple tokens in parallel. This is the drafter model, which must be paired with google/gemma-4-31B-it.
Quick Start
Installation
uv pip install -U --torch-backend=auto \
"vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/41703/head"SGLang:
uv pip install "git+https://github.com/sgl-project/sglang.git@refs/pull/23000/head#subdirectory=python"Launch Server
vLLM:
vllm serve google/gemma-4-31B-it \
--speculative-config '{"method": "dflash", "model": "z-lab/gemma-4-31B-it-DFlash", "num_speculative_tokens": 15, "attention_backend": "flash_attn"}' \
--attention-backend triton_attn \
--max-num-batched-tokens 32768 \
--trust-remote-codeSGLang:
# Optional: enable schedule overlapping (experimental, may not be stable)
# export SGLANG_ENABLE_SPEC_V2=1
# export SGLANG_ENABLE_DFLASH_SPEC_V2=1
# export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
python -m sglang.launch_server \
--model-path google/gemma-4-31B-it \
--speculative-algorithm DFLASH \
--speculative-draft-model-path z-lab/gemma-4-31B-it-DFlash \
--speculative-num-draft-tokens 16 \
--tp-size 1 \
--attention-backend triton \
--speculative-draft-attention-backend fa4 \
--trust-remote-codeUsage
For vLLM, use port 8000. For SGLang, use port 30000.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[{"role": "user", "content": "Write a quicksort in Python."}],
max_tokens=4096,
temperature=0.0,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.content)Benchmark Results
Setup: Single NVIDIA B300 GPU per server/run, vLLM, thinking enabled, max output length 4096, greedy decoding.
Throughput and Speedup
DFlash achieves up to 5.8x speedup at concurrency 1.
Generated tokens/sec (speedup vs. autoregressive baseline)
Block Size = 16
| Task | Concurrency | AR | DFlash |
|---|---|---|---|
| Math500 | 1 | 77 | 447 (5.8x) |
| 8 | 511 | 2650 (5.2x) | |
| 32 | 1308 | 4962 (3.8x) | |
| GSM8K | 1 | 78 | 408 (5.3x) |
| 8 | 520 | 2321 (4.5x) | |
| 32 | 1382 | 4447 (3.2x) | |
| HumanEval | 1 | 76 | 420 (5.6x) |
| 8 | 494 | 2389 (4.8x) | |
| 32 | 1145 | 4139 (3.6x) | |
| MBPP | 1 | 79 | 343 (4.4x) |
| 8 | 535 | 2036 (3.8x) | |
| 32 | 1389 | 3636 (2.6x) | |
| MT-Bench | 1 | 79 | 236 (3.0x) |
| 8 | 503 | 1334 (2.7x) | |
| 32 | 1177 | 2257 (1.9x) |
Acceptance Length
| Task | c1 | c8 | c32 |
|---|---|---|---|
| Math500 | 8.59 | 8.59 | 8.62 |
| GSM8K | 7.53 | 7.50 | 7.52 |
| HumanEval | 8.00 | 7.89 | 7.96 |
| MBPP | 6.13 | 6.13 | 6.14 |
| MT-Bench | 4.23 | 4.19 | 4.19 |
Acknowledgements
Special thanks to David Wang for his outstanding engineering support on this project. We are also grateful to Modal, InnoMatrix, and Yotta Labs for providing the compute resources used to train this draft model.
Citation
If you find DFlash useful, please cite our work. To share feedback on DFlash or request new model support, please fill out this form: DFlash Feedback.
@article{chen2026dflash,
title = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
author = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
journal = {arXiv preprint arXiv:2602.06036},
year = {2026}
}