docs
// Tools

erectl models

Full model lifecycle from the terminal. List, inspect, update every metadata field, version, delete, share to the catalog, revalidate, probe capabilities, and ask the router for workspace-fitted recommendations. One flag per action; bulk delete the only multi-id verb.

// requires an active erectl deployment and an API key with inference scope.

Overview#

The erectl models command group provides full lifecycle management for models stored in your project. It follows the unified command pattern used across all erectl resource commands:

Pattern
erectl models # List all models erectl models <MODEL_ID> # Show model details erectl models <MODEL_ID> --<action> # Perform an action on a model erectl models update <MODEL_ID> [options] # Update metadata erectl models hf-overrides <MODEL_ID> # Show HF config overrides erectl models versions <MODEL_ID> # Manage model versions

Only one action flag per invocation, and multiple IDs only with --delete (bulk). Metadata changes go through erectl models update.

Back to erectl CLI overview.

List Models#

// try this first Run erectl models with no arguments to print the table of IDs you can substitute into every other example on this page. Append --help to any subcommand for the in-CLI reference.

Omit all IDs to list models in your project. The default table output includes ID, Name, Format, Size, Status, Version, and Shared columns.

bash
# Basic listing erectl models # Wide table output (adds Architecture, Quantization, Context, Parameters, Created) erectl models -o wide # Request extended fields on the JSON response (no effect on text tables) erectl models --extended -o json # Paginate results erectl models --limit 10 --offset 20 # JSON output erectl models -o json

Listing Options

Option Description
--limit <n> Maximum number of results. The router applies a server-side ceiling of 100 (and uses 100 as the fallback when no value is supplied); larger values are accepted by the CLI but silently truncated to 100 by the API.
--offset <n> Pagination offset. When more results exist, a hint is printed at the end of the table.
--extended Request extended fields (architecture, quantization, context length, parameter count, creation time) from the API. Only changes the response payload when combined with -o json; the default text table is unaffected. To see the extra columns in a table, use -o wide.

Get Model#

Provide a single model ID to display its full details.

bash
# Human-readable key-value output erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 # Show full description without truncation erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --full # JSON output erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 -o json

The detail view includes ID, Name, Created, Owner, Format, Size, Status, Version, Is Latest, Description, Architecture, Quantization, Context Length, Parameters, Workload Type, Shared, Catalog Role, Origin, Trust Remote Code, License, Hugging Face ID, Capabilities, the two parsers, Layers, Hidden Size, Vocab Size, Torch Dtype, the expert counts, Pre-Quantized, Reasoning Efforts, Engine Tuning (the settings that are not auto, or auto), and, for a model you own, Speculative Decoding and the number of HF config override rows.

The Capabilities line states whether structured output is proven by probe or not proven: only the capability probe sets that tag, and a model the probe found without it refuses a json_schema request. json_object carries no schema and is accepted on every chat model.

The description is truncated to 120 characters by default. Use --full to display it in its entirety and to print the assembled HF config overrides.

Update Metadata#

erectl models update <MODEL_ID> changes only the fields you pass; every other field keeps its value. none clears a field, and the parsers reset with auto. A value the server refuses is named in the error, and nothing of the call is stored.

bash
# Update description and context length erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 \ --description "Fine-tuned for code generation" \ --context-length 8192 # Set the capability tags and the parsers erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 \ --capabilities chat,vision \ --tool-call-parser hermes \ --reasoning-parser auto # Update the dimensions, then clear one that was wrong erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 \ --parameter-count 7000000000 --hidden-size 4096 --num-layers 32 --vocab-size 32000 erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --vocab-size none # Mark as pre-quantized erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 \ --pre-quantized \ --pre-quantization-method awq \ --pre-quantization-bits 4 # Tune the vLLM launch, then return one setting to auto erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 \ --max-num-seqs 128 --gpu-memory-utilization 0.9 --prefix-caching off erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --prefix-caching auto # Print the request body without sending it erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --license MIT --dry-run

Engine settings are the fields an engine launches with: quantization, context length, the parsers, trust remote code, the engine tuning, capabilities, the dimensions except the vocabulary size, the torch dtype, the expert counts, the pre-quantization fields, speculative decoding and the HF config overrides. Changing one needs an API key with the management scope, and every endpoint serving the model is relaunched with the new value. Passing a setting's current value changes nothing and needs no extra scope. Changing trust remote code also needs the project owner.

A field you set is kept across revalidation: Revalidate re-reads the model's config.json, README and LICENSE only for the fields you have not edited. Clearing a field with none hands it back: the next revalidation fills it from those files again.

Listing Options

Option Type Description
--description <text> string Markdown description of the model. "" clears it.
--description-file <path|-> path Read the description from a file, or from standard input with -.
--license <text|none> string License identifier (e.g., "MIT", "apache-2.0").
--hugging-face-id <owner/name|none> string Upstream Hugging Face repository the weights come from.
--workload-type <tag,...> string Workload tags: chat, code, reasoning, embedding, multilingual. At least one.
--reasoning-efforts <detected|unsupported|list> string reasoning_effort values the model accepts: detected (reset to the values read from the model's reasoning parser, chat template and family), unsupported (the model does not reason), or a comma-separated list. default is accepted as another name for detected.
--architecture <text|none> string Model architecture (e.g., "qwen3").

Engine Options

Option Type Description
--capabilities <tag,...|none> string Capability tags, replacing the whole list: chat, vision, video_input, audio_input, transcription, speech, realtime, forced_tool_choice. Requests are admitted per tag. structured_output is set only by the capability probe and is kept whatever you send.
--quantization <method|native> string Runtime quantization method applied by the serving backend. native serves the weights as stored.
--context-length <n> integer Maximum context length in tokens (vLLM --max-model-len), at least 128. It cannot be cleared.
--tool-call-parser <name|auto> string vLLM --tool-call-parser. auto detects it from the architecture.
--reasoning-parser <name|auto> string vLLM --reasoning-parser. auto detects it from the architecture.
--[no-]trust-remote-code flag Whether the engine may execute code shipped in the model bundle. Project owners only.
--scheduling-policy <priority|fcfs> string vLLM --scheduling-policy. priority (the default) serves realtime and interactive requests first; fcfs serves requests in arrival order.
--kv-cache-dtype <dtype> string vLLM --kv-cache-dtype. auto (the default) passes no flag; any other value is passed as given. An endpoint can override it.
--tool-narrowing <default|on|off> string Chat tool narrowing for the model. on starts its chat turns with a core of tools and a catalog the model loads the rest from; off never narrows it; default follows the platform setting. See Chat Options.
--steering-role <default|user_note> string How the router's mid-turn corrections reach the model. user_note sends them as a platform note after the tool results, which keeps the prompt cache; default sends system instructions. Refused, with the reason, on a model whose chat template does not take a note. erectl models show prints both settings.
--torch-dtype <dtype|none> string Native PyTorch dtype: float16, bfloat16, or float32.
--[no-]pre-quantized flag Whether the weights are stored in quantized form.
--pre-quantization-method <m|none> string Pre-quantization algorithm: gptq, awq, compressed-tensors, or fp8.
--pre-quantization-bits <4|8|none> integer Bit width of the pre-quantization.

Dimension Options

Option Type Description
--parameter-count <n|none> integer Total number of model parameters; sizes the VRAM estimate.
--hidden-size <n|none> integer Hidden layer dimensionality.
--num-layers <n|none> integer Number of transformer layers.
--vocab-size <n|none> integer Vocabulary size.
--num-experts <n|none> integer Total number of MoE experts.
--num-experts-per-tok <n|none> integer Number of experts activated per token (MoE models).

Engine Tuning Options

vLLM launch settings for every engine serving the model. Each is auto unless set: the Erebine Inference Microservice (EIM) chooses the value as it does for any model. auto and none both return a setting to auto. A value the operator sets on an EIM, by option, environment variable or vLLM argument, wins over the model's on that EIM.

Option Type Description
--expert-parallel <on|off|auto> string vLLM --enable-expert-parallel for a mixture-of-experts model on more than one GPU. Auto turns it on there.
--enable-eplb <on|auto> string vLLM --enable-eplb, expert parallel load balancing. Applies with expert parallel across more than one rank on NVIDIA CUDA. Refused with --expert-parallel off.
--all2all-backend <backend|auto> string vLLM --all2all-backend under expert parallel: allgather_reducescatter, flashinfer_nvlink_two_sided or flashinfer_nvlink_one_sided.
--gpu-memory-utilization <fraction|auto> number vLLM --gpu-memory-utilization, above 0 and at most 1.
--max-num-seqs <n|auto> integer vLLM --max-num-seqs, 1 or more.
--max-num-batched-tokens <n|auto> integer vLLM --max-num-batched-tokens, 1 or more and at least --max-num-seqs when both are set.
--long-prefill-token-threshold <n|auto> integer vLLM --long-prefill-token-threshold, 0 or more; 0 is no cap. A value above the context length is not sent.
--enforce-eager <on|auto> string vLLM --enforce-eager: run without CUDA graphs.
--prefix-caching <on|off|auto> string vLLM automatic prefix caching. Auto keeps it on; audio models always run without it.
--async-scheduling <off|auto> string vLLM async scheduling. off passes --no-async-scheduling. A model cannot force it on.
--disable-custom-all-reduce <on|auto> string vLLM --disable-custom-all-reduce: every all-reduce across GPUs uses NCCL.
--engine-config <flag=json|flag=none> string One vLLM JSON config: attention_config, compilation_config, speculative_config or reasoning_config, set to one JSON object, which is merged over the object the EIM derives for that flag (a key set to null removes the EIM's value). flag=none clears that flag. Repeat it; flags left out keep their objects. A key that no vLLM release the EIM images ship accepts is refused.
--clear-engine-configs flag Clear every vLLM JSON config.
bash
# Set one vLLM JSON config, then clear it erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --engine-config attention_config='{"indexer_kv_dtype":"mxfp4"}' erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --engine-config attention_config=none

HF Config Override Options

Option Type Description
--hf-override <key.path=type:value> string One override row; type is string, number or boolean. Repeat it. The rows replace the whole set.
--hf-overrides-file <rows.json> path Replace the set with the rows in a file, in the shape erectl models hf-overrides -o json prints.
--clear-hf-overrides flag Remove every override row.

Overrides and an enabled speculative decoding setting refuse each other. One call may clear the overrides and enable speculation; it may not set both.

File Options

Option Type Description
--metadata <file.json|-> path Apply a whole update body from a file (see Metadata File). A field set both in the file and by an option is refused.

Speculative Decoding Options

These change the model's speculative decoding setting. Only the options you pass change; the rest of the setting is kept. The server checks the result as a whole and refuses a setting that breaks a rule, naming it. The setting, its methods and its rules are described in Model Management.

Option Type Description
--speculative on|off string Speculate by default on every endpoint of the model. on needs a method.
--speculative-method <method>|none string mtp, ngram, eagle, eagle3, dflash, draft_model, medusa, or mlp_speculator. none clears it.
--speculative-tokens <n>|default integer Tokens drafted per step, 1 to 32. default uses the method's default.
--draft-model <id>|none string Id of the catalog model that drafts. Required by eagle, eagle3, dflash, draft_model, medusa and mlp_speculator; refused by mtp and ngram. none clears it.
--ngram-fallback on|off string mtp only: speculate with ngram when the checkpoint ships no multi-token-prediction layers.
--speculative-temperature <x>|none number Default temperature (0 to 2) for requests that send none while the model speculates.
--speculative-top-p <x>|none number Default top_p (above 0, at most 1) for requests that send none while the model speculates.
--speculative-top-k <n>|none integer Default top_k (1 or more) for requests that send none while the model speculates. A top_k default lets a thinking budget run under speculative decoding; see Sampling defaults.
--speculative-reset flag Reset the whole setting: off, no method, no draft, no sampling defaults. Takes no other speculative option.

The model details (erectl models <id>) show the setting on the Speculative Decoding line.

Metadata File#

A metadata file is the body of PATCH /v1/models/{id}: a JSON object with snake_case keys, where an absent key leaves the field alone and null clears it. A key the request body does not define is refused by name, so a typo never goes unnoticed. The same file works with erectl models update --metadata and with erectl upload --metadata.

A speech model's voice settings ride under speech: an object with disabled_voices, default_voice and aliases. Absent leaves the settings alone, null clears them, and an object replaces them whole. A voice the model does not offer is refused by name; see Speech Voices.

speech.json
{ "speech": { "disabled_voices": ["uncle_fu"], "default_voice": "serena", "aliases": {"alloy": "serena", "echo": "ryan"} } }
model.json
{ "description": "Qwen3 8B instruct, fine-tuned for code.", "license": "apache-2.0", "workload_type": "chat,code", "capabilities": ["chat", "vision"], "context_length": 32768, "tool_call_parser": "hermes", "reasoning_efforts": ["low", "medium", "high"], "torch_dtype": null, "hf_overrides": [ {"key_path": "rope_scaling.factor", "value_type": "number", "value": "4.0"} ] }
bash
# Apply the file erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --metadata model.json # Read it from standard input, and add one option the file does not set cat model.json | erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --metadata - --no-trust-remote-code

HF Config Overrides#

erectl models hf-overrides <MODEL_ID> lists the Hugging Face config override rows stored on a model you own. With --preview it assembles the rows in a file into the document the engine receives as --hf-overrides, and stores nothing.

bash
# List the rows erectl models hf-overrides 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 # Save them, edit the file, preview it, then store it erectl models hf-overrides 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 -o json > rows.json erectl models hf-overrides 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --preview rows.json erectl models update 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --hf-overrides-file rows.json
Option Description
--preview <rows.json|-> Assemble the rows in a file ({"rows": [...]}) without storing them. A set the server refuses is named in the error.

Delete a Model#

Delete a single model or multiple models in bulk. A confirmation prompt is shown unless --force is supplied. Use --dry-run to preview the operation without making changes.

bash
# Delete with confirmation prompt erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --delete # Skip confirmation erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --delete --force # Bulk delete (multiple IDs, only supported with --delete) erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 2a3c5b6e-7d8f-4019-9c2a-1b4d3e5f6a7c 3f7e8d9a-1c2b-4d5e-8f60-7a8b9c0d1e2f --delete --force # Preview without deleting erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --delete --dry-run

Bulk deletion prints a per-model result ([OK] / [FAIL]) and a summary line showing how many succeeded. With -o json, one id or several print the same document: an array with {"id", "status"} per id, in argument order, where status is deleted or failed. The command exits 1 if any model was not deleted, and the reason goes to standard error.

A model that another model uses as its speculative decoding draft is not deleted: the server answers 409 naming those models. Change or clear their --draft-model first.

// see also REST: DELETE /v1/models/{id}

Share / Unshare#

Share a model to the project catalog so it can be discovered and deployed by other users. Use --unshare to remove it from the catalog. Use --catalog-info to inspect the current catalog state.

bash
# Share to catalog erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --share # Remove from catalog erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --unshare # Show catalog info (role, deployment count, public flag, featured flag, shared date) erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --catalog-info

Catalog info displays: ID, Catalog Role, Deployment Count, On Shared EIMs, Public, Featured, and Shared At.

Revalidate#

Trigger a re-check of the model's files and update its validation status. This is useful after a file has been corrected or a partial upload completed.

bash
# Queue revalidation and return immediately erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --revalidate # Queue revalidation and poll until a terminal status is reached erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --revalidate --wait # Set a custom wait timeout (default: 600 seconds) erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --revalidate --wait --wait-timeout 300

Revalidation Options

Option Description
--revalidate Trigger model revalidation. Requires a single model ID.
--wait Poll until revalidation reaches a terminal state (active, ready, validated, error, failed, or invalid). Only valid with --revalidate.
--wait-timeout <seconds> Maximum seconds to wait when --wait is used. Default: 600.

Version Management#

The models versions subcommand manages semantic versions for a model. Versions track deployment history and allow rollback to a previous state.

bash
# List all versions for a model erectl models versions 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 # Create a new version (semver string required) erectl models versions 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --create 2.0.0 erectl models versions 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --create 2.0.0 --notes "Improved accuracy on code tasks" # Promote a version to latest erectl models versions 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --promote 2.0.0 # Rollback to a previous version erectl models versions 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --rollback 1.0.0

Version Options

Option Description
--create <version> Create a new version using a semver string (e.g., 2.0.0).
--promote <version> Mark the given version as the latest active version.
--rollback <version> Roll back to a previously created version.
--notes <text> Version release notes. Only valid with --create.

Only one action (--create, --promote, or --rollback) may be specified per invocation.

The version list table shows: Version, ID, Latest, Status, Notes, and Created.

// see also Model Versioning for promotion semantics and rollback safety.

Options Reference#

Full reference for all flags and options accepted by erectl models. The options of erectl models update are listed under Update Metadata, including the speculative decoding options and the engine tuning options.

Action Flags (mutually exclusive)

Flag State Requires ID Description
--delete destructive Yes (one or more) Delete one or more models. Prompts for confirmation unless --force is set.
--share reversible Yes (one) Share the model to the catalog.
--unshare reversible Yes (one) Remove the model from the catalog.
--revalidate safe Yes (one) Trigger model file revalidation.
--catalog-info read-only Yes (one) Display catalog information for the model.

Modifier Flags

Flag Description
--force Skip the confirmation prompt on destructive operations.
--full Show the full description instead of truncating to 120 characters, and the assembled HF config overrides.
--extended Request extended fields on the API response. Affects JSON output only; use -o wide to render extra columns in the text table.
--wait Poll until revalidation completes. Only valid with --revalidate.
--wait-timeout <n> Timeout in seconds for --wait. Default: 600.
--limit <n> Maximum number of results when listing. The router enforces a server-side ceiling of 100 and uses 100 as the fallback when omitted.
--offset <n> Pagination offset when listing.

Examples#

Update and Revalidate a Model

bash
MODEL_ID="8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01" # Set metadata erectl models update "$MODEL_ID" \ --description "Qwen3 7B instruct, fine-tuned for code" \ --architecture "qwen3" \ --context-length 32768 \ --parameter-count 7000000000 \ --workload-type "chat,code" # Revalidate and wait for result erectl models "$MODEL_ID" --revalidate --wait

Speculative Decoding with a Draft Model

bash
MODEL_ID="8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01" DRAFT_ID="2a3c5b6e-7d8f-4019-9c2a-1b4d3e5f6a7c" # DFlash with 7 speculative tokens; the EIM syncs the draft itself erectl models update "$MODEL_ID" \ --speculative on \ --speculative-method dflash \ --speculative-tokens 7 \ --draft-model "$DRAFT_ID" # Let thinking budgets run while speculating erectl models update "$MODEL_ID" --speculative-top-k 20 # Native MTP, falling back to n-gram when the checkpoint has no MTP layers erectl models update "$MODEL_ID" --speculative-method mtp --draft-model none --ngram-fallback on # Turn it off, keeping the rest of the setting; or reset it entirely erectl models update "$MODEL_ID" --speculative off erectl models update "$MODEL_ID" --speculative-reset

Version Promotion Workflow

bash
MODEL_ID="8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01" # Create new version after fine-tuning erectl models versions "$MODEL_ID" \ --create 2.1.0 \ --notes "Reduced hallucination rate on factual queries" # Verify the version list erectl models versions "$MODEL_ID" # Promote once tested erectl models versions "$MODEL_ID" --promote 2.1.0

Bulk Delete Old Models

bash
# Preview first erectl models 4a5b6c7d-8e9f-4012-9a3b-5c6d7e8f9012 5b6c7d8e-9f01-4234-8a4b-6c7d8e9f0123 6c7d8e9f-0123-4456-8b5c-7d8e9f012345 --delete --dry-run # Execute erectl models 4a5b6c7d-8e9f-4012-9a3b-5c6d7e8f9012 5b6c7d8e-9f01-4234-8a4b-6c7d8e9f0123 6c7d8e9f-0123-4456-8b5c-7d8e9f012345 --delete --force

Share to Catalog and Check Info

bash
erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --share erectl models 8b1f4c2d-9e7a-4f3b-bc11-2d8e6a4c5f01 --catalog-info