Deploy on GPUs: workstation, HPC, Docker¶

polyjev itself is small and CPU-only. The GPU work happens in the model server (vLLM) or, with the hf provider, inside your own process. The usual layouts:

laptop / app ──HTTP──▶ polyjev :8011 ──▶ vLLM :8000 on the GPU box    (open models)
                              └──────▶ Claude / OpenAI / Gemini APIs  (hosted models)

Which model for which GPU¶

Any chat model vLLM serves works. These are good starting points (bf16 unless noted; leave room for the KV cache):

GPU memory	examples	text model	vision model
16–24 GB	RTX 4080/4090, 3090, A5000, L4	`Qwen/Qwen3-4B-Instruct-2507` (8 GB)	`Qwen/Qwen3-VL-8B-Instruct` (tight on 24 GB)
32 GB	RTX 5090, RTX 5000 Ada	`Qwen/Qwen3-8B` (thinking off)	`google/gemma-4-12B-it`, `Qwen/Qwen3-VL-8B-Instruct`
48 GB	RTX 6000 Ada, A6000, L40S	`Qwen/Qwen3-30B-A3B-Instruct-2507-FP8` (MoE, fast)	`Qwen/Qwen3-VL-8B-Instruct` with a long context
80–96 GB	A100/H100 80 GB, RTX PRO 6000	`google/gemma-4-31B-it`	`google/gemma-4-26B-A4B-it` (MoE), `google/gemma-4-31B-it`
4–8 GPUs	a full node	`Qwen/Qwen3-235B-A22B-Instruct-2507-FP8` (`TP=4` on 80 GB cards)

Instruct models that answer directly (no thinking phase) suit logprobs reads best. For thinking models, keep enable_thinking: false (the recipes below do).

Driver versions

vLLM 0.30's default build needs an R580+ driver (CUDA 13). On older drivers (CUDA 12.x) use the +cu129 build: deploy/slurm/setup-env.sh picks it automatically, and for Docker set VLLM_TAG=v0.30.0-cu129.

RTX workstation (Docker)¶

Needs Docker and the NVIDIA Container Toolkit.

git clone https://github.com/PraveenAShukla/polyjev && cd polyjev/deploy
cp .env.example .env          # set MODEL, VLLM_TAG, and API keys for any hosted models
docker compose --profile gpu up -d
docker compose logs -f vllm   # first start downloads the model
../scripts/smoke.sh http://localhost:8011

Open http://localhost:8011 for the playground. Without --profile gpu, only the polyjev container starts, which is useful when vLLM runs elsewhere (VLLM_BASE_URL) or you only use hosted models (POLYJEV_DEFAULT_MODEL=claude).

HPC cluster (SLURM)¶

Two recipes live in deploy/slurm/:

setup-env.sh (once): a venv with vLLM + polyjev, and the model downloaded to a shared cache. It detects the driver's CUDA version and picks the matching vLLM/torch build.
polyjev.sbatch (every time): one job runs vLLM and polyjev on the same GPU node, writes a config, runs polyjev probe, and prints the SSH tunnel command.

# on the login node, in your checkout
srun -p <gpu-partition> --gres=gpu:1 -c 8 --mem=32G -t 1:00:00 \
  env ENV_DIR=$HOME/polyjev-env MODEL=Qwen/Qwen3-8B bash deploy/slurm/setup-env.sh

source $HOME/polyjev-env/bin/activate
sbatch -p <gpu-partition> --export=ALL,MODEL=Qwen/Qwen3-8B deploy/slurm/polyjev.sbatch
tail -f polyjev-<jobid>.log      # wait for "Serving ... From your laptop: ssh -N -L ..."

On your laptop, run the printed tunnel, then:

scripts/smoke.sh http://localhost:8011
python -c 'import polyjev as pj; print(pj.Remote("http://localhost:8011").decide("The server room is on fire", {"urgent": pj.Noul("Is this urgent?")}))'

Disk space

The venv is about 10 GB and an 8B model about 16 GB. Cluster home directories often have small quotas, so point ENV_DIR and HF_HOME at project or Lustre storage, for example ENV_DIR=/l/users/$USER/polyjev/venv and HF_HOME=/l/users/$USER/polyjev/hf-cache, and pass the same HF_HOME to sbatch.

Settings (environment variables, pass them with sbatch --export=ALL,NAME=value):

variable	default	meaning
`MODEL`	`Qwen/Qwen3-8B`	Hugging Face model id
`TP`	GPUs on the node	tensor parallel size (`--gres=gpu:4` → 4)
`GPU_UTIL`	`auto`	vLLM `--gpu-memory-utilization`; `auto` takes what is free on the job's GPUs at start (at most 0.90), so a display does not break startup. Below 20% free the job stops with a message: something else holds the GPU (resubmit with `-x <node>`)
`MAX_MODEL_LEN`	`8192`	context length (prompt + answer)
`VLLM_SIF`	empty	an Apptainer/Singularity image of `vllm/vllm-openai` instead of the venv
`CHAT_TEMPLATE_KWARGS`	`{"enable_thinking": false}`	passed on every request
`POLYJEV_API_KEY`	empty	require a bearer token
`HF_HOME`	`~/.cache/huggingface`	model cache (keep it on shared storage)
`EXTRA_VLLM_ARGS`	empty	anything else for `vllm serve`
`VLLM_USE_FLASHINFER_SAMPLER`	`0`	FlashInfer's sampler compiles CUDA code with `nvcc` on first use; off so nodes without a CUDA toolkit work

Containers instead of a venv:

singularity pull $HOME/vllm.sif docker://vllm/vllm-openai:v0.30.0-cu129
sbatch --export=ALL,VLLM_SIF=$HOME/vllm.sif,POLYJEV_CMD="uvx --from polyjev[server] polyjev" deploy/slurm/polyjev.sbatch

No server at all: inside any GPU job, load the model in-process with the hf provider. This is the most exact option, since it reads label logits directly, and suits batch jobs:

judge = pj.Polyjev(config={"models": {"m": {"provider": "hf", "model": "Qwen/Qwen3-4B-Instruct-2507",
                                            "options": {"device_map": "cuda", "dtype": "bfloat16"}}}}, model="m")
results = judge.decide_many(states, questions, concurrency=1)

Security¶

The server binds 127.0.0.1 by default. --host 0.0.0.0 exposes it: set POLYJEV_API_KEY (or server.api_key) when you do.
Only models in your config are served. server.allow_adhoc_models: true lets clients name any provider/model, including hosted ones billed to your keys.
API keys stay in the server's environment; clients never see them.