docs
// Infrastructure

EIM on Windows

On Windows, the Erebine Inference Microservice (EIM) runs in a container inside a WSL2 distribution. WSL2 has no Linux GPU driver: every GPU is WSL's /dev/dxg device, backed by the Windows driver. NVIDIA and AMD GPUs run there; erectl deploy eim detects WSL and deploys the shape each vendor needs.

platform
Windows 11, a WSL2 distribution with systemd
GPU device
/dev/dxg (no /dev/kfd, no /dev/nvidia*)
runs while
the WSL distribution runs

What Runs

HardwareUnder WSL2How
nvidia-gpuRunsThe NVIDIA container toolkit's WSL mode hands the container /dev/dxg and the Windows driver's libraries as one device, nvidia.com/gpu=all. More than one EIM shares the GPUs; MIG and one GPU per EIM are not available.
amd-gpuRuns, one GPU/dev/dxg, Microsoft's libdxcore.so mounted read-only, unprivileged, PyTorch's expandable segments off. The ROCm in eim-vllm-rocm carries AMD's WSL bridge library.
intel-gpuRefusedNo WSL2 path for Intel GPUs. Run the Intel EIM on native Linux.
cpu, amd-cpuRunsAs on native Linux.

eim-vllm-rocm carries kernels for gfx1200 and gfx1201 (RDNA 4) and gfx1150 to gfx1153 (Ryzen AI APUs). AMD also supports RDNA 3 GPUs (gfx1100 to gfx1102) under WSL2; the image has no kernels for them.

On Windows

  • Windows 11 and a current WSL: run wsl --update. The distribution must run under WSL2 (wsl --set-version <distribution> 2).
  • The current GPU driver from the GPU vendor. For AMD, AMD Software: Adrenalin Edition or PRO Edition.
  • The WSL VM's memory and CPUs are the EIM's. Set them in %UserProfile%\.wslconfig, under [wsl2]: memory= (half of Windows' memory by default) and processors=.
  • WSL stops a distribution that has been idle for instanceIdleTimeout, 15 seconds by default. To keep the EIM running when no terminal is open, set it under [general]:
[general]
instanceIdleTimeout=-1

Run wsl --shutdown after changing .wslconfig. Windows does not start a distribution at boot; the EIM starts when the distribution does.

In the Distribution

  • systemd, for --service. Add to /etc/wsl.conf, then run wsl --shutdown on Windows:
[boot]
systemd=true
  • podman with podman-compose, or docker with the compose plugin.
  • NVIDIA: the NVIDIA container toolkit, which generates its WSL-mode device spec itself.
  • AMD: /dev/dxg and /usr/lib/wsl/lib/libdxcore.so, which WSL provides. No ROCm is needed in the distribution. With ROCm installed there, --auto-hardware-detect finds the GPU through its amd-smi; without it, pass --hardware-type amd-gpu.

Deploy with erectl

sudo erectl deploy eim --hardware-type amd-gpu --service --region us-east

The plan, printed before every deploy and by --dry-run, says the host is WSL2 and what that changes. On WSL2 the command:

  • refuses, before it changes anything, an AMD GPU without /dev/dxg, without /usr/lib/wsl/lib/libdxcore.so, or on a kernel that is not a WSL2 kernel, naming what is missing and what to do;
  • refuses --service when systemd is not running, with the /etc/wsl.conf lines;
  • refuses more than one AMD EIM, --gpu-link and --no-sysctl for an AMD GPU, --mig and a pinned layout of more than one NVIDIA EIM, and --hardware-type intel-gpu;
  • installs a unit that waits for /dev/dxg (and, for AMD, libdxcore.so) before it starts the EIM.

See erectl deploy for every flag.

Deploy with Compose

From the erebine/container-agents repository, inside the distribution:

git clone https://github.com/erebine/container-agents.git
cd container-agents/compose
sudo mkdir -p /data/erebine/models /data/erebine/config /data/erebine/vllm-cache
sudo chown -R 5152:5152 /data/erebine
export EREBINE_EIM_JOIN_KEY=ejk_your_key_here
sudo -E podman compose -f compose.agent-amd-rocm-wsl.yaml up -d

compose.agent-amd-rocm-wsl.yaml is the AMD stack for WSL2. Its header comments explain each line that differs from compose.agent-amd-rocm.yaml, which does not run under WSL2. An NVIDIA GPU uses compose.agent-nvidia.yaml unchanged.

What Differs from Native Linux

  • GPU memory is shared with Windows. The desktop holds part of it, and its use moves. The EIM sizes the engine's GPU memory from what is free when a model launches, at most 85% of the GPU. EREBINE_EIM_WSL_HEADROOM=off turns that off.
  • Readings. On AMD, amd-smi reports the GPU's model, architecture, memory, activity, power and temperature, and no PCIe link, firmware, NUMA node or RAS figures; per-process memory is not reported. The EIM needs none of the missing ones and binds no NUMA node.
  • Allocator. On AMD, PyTorch's expandable segments are off, so GPU memory can fragment more under long, mixed load.
  • Lifetime. The EIM runs while the WSL distribution runs, not from Windows boot. See On Windows for instanceIdleTimeout.
  • Host settings. Kernel command-line changes go in kernelCommandLine under [wsl2] in .wslconfig; erectl prints the change and never writes that file.

Troubleshooting

  • erebine-agents: /dev/dxg did not appear within 120 s in journalctl -u erebine-agents: the distribution has no GPU device. Run wsl --update, install the current GPU driver, run wsl --shutdown. Check ls -l /dev/dxg.
  • erebine-agents: /usr/lib/wsl/lib/libdxcore.so did not appear within 120 s: WSL has not mounted the Windows GPU libraries. The same steps; check ls /usr/lib/wsl/lib.
  • erebine-agents: /dev/kfd did not appear within 120 s on WSL2: the unit was written for native Linux, by an erectl that did not detect WSL. Run the deploy again with the current erectl.
  • [DxcoreLoader] Cannot load libdxcore.so, or hsa_init failing, in the EIM's log: the container lacks libdxcore.so on its loader path. Mount /usr/lib/wsl/lib/libdxcore.so at /usr/lib64/libdxcore.so; mounting /usr/lib/wsl alone is not enough.
  • CUDA error: invalid argument on the first GPU allocation of an AMD EIM: PyTorch's expandable segments are on. Set PYTORCH_ALLOC_CONF=expandable_segments:False.
  • --service on WSL: this distribution runs without systemd: turn systemd on in /etc/wsl.conf (see In the Distribution).
  • GPU image found no GPU it can drive: the container was started without /dev/dxg, or for AMD without the libdxcore.so mount. The log line names what detection found.