vLLM-Omni extends vLLM with omni-modality inference (audio, video, image). It runs as a Podman container, serving an OpenAI-compatible TTS API on port 8091.
The Qwen3-TTS family has 5 model variants across 3 task types. Each instance serves exactly one model — to run multiple simultaneously, use separate containers on different ports.
| Model | Task | Size |
|---|---|---|
Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice |
Predefined speaker voices | 1.7B |
Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice |
Predefined speaker voices | 0.6B |
Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign |
Describe voice in natural language | 1.7B |
Qwen/Qwen3-TTS-12Hz-0.6B-VoiceDesign |
Describe voice in natural language | 0.6B |
Qwen/Qwen3-TTS-12Hz-1.7B-Base |
Voice cloning from reference audio | 1.7B |
The task_type field in API requests must match the
loaded model (CustomVoice, VoiceDesign, or
Base).
Create a Containerfile that layers
vllm-omni on top of the official vLLM image:
FROM vllm/vllm-openai:v0.20.0
RUN pip install vllm-omni==0.20.0rc1
RUN pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.4/flash_attn-2.8.3+cu130torch2.11-cp312-cp312-linux_x86_64.whl"
ENTRYPOINT ["vllm-omni"]The flash-attn line installs Flash Attention 2 from a
prebuilt wheel — no compilation needed. The wheel must match the CUDA,
PyTorch, and Python versions inside the base image. To find yours:
Query the base image versions:
podman run --rm --entrypoint python3 vllm/vllm-openai:v0.20.0 \
-c "import torch, sys; print(f'CUDA: {torch.version.cuda} Torch: {torch.__version__} Python: {sys.version.split()[0]}')"Browse the mjun0812/flash-attention-prebuild-wheels
releases (wheels built via GitHub Actions from source). Find the
.whl file matching your
cu{CUDA}torch{TORCH}-cp{PYTHON}-cp{PYTHON}-linux_x86_64
pattern and replace the URL above.
Build the image:
podman build -t vllm-omni .Each instance serves exactly one model — stop one before starting
another on the same port. The task_type in API requests
must match the loaded model.
The -v ~/.cache/huggingface mount reuses
already-downloaded model weights across container runs. The API is
available at http://0.0.0.0:8091 (not localhost — Podman
networking quirk).
All run commands below include these performance flags:
| Variable | Purpose |
|---|---|
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True |
Reduces CUDA allocator fragmentation, helping Code2Wav graph capture succeed when free VRAM is tight |
TORCH_FLOAT32_MATMUL_PRECISION=high |
Enables TF32 matmuls in PyTorch on Ampere+ GPUs (~10–20% faster) |
NVIDIA_TF32_OVERRIDE=1 |
Forces TF32 at the CUDA driver level (belt-and-suspenders with the above) |
vLLM-Omni’s 2-stage architecture (Talker + Code2Wav) spawns two independent engines. By default each pre-allocates a large KV cache pool, which wastes VRAM on TTS workloads.
--gpu-memory-utilization 0.08 — minimizes the global KV
cache pool--stage-overrides '{"0":{...},"1":{...}}' — per-stage
caps on memory and concurrency--max-num-seqs 2 — lets the next sentence’s request
prefill while the current one is still generating, which matters for
chunked TTS streaming (sentence pipelining). The KV pool is shared, so
going from 1 to 2 costs almost no extra VRAMFor a 12 GB GPU sharing VRAM with an LLM
(e.g. Ollama), the 1.7B model fits in ~8 GB with these flags, leaving
room for a ~2–3 GB LLM. Without them, vLLM defaults to
gpu_memory_utilization=0.9 and will consume nearly all
VRAM.
Given that you have 24+ GB of VRAM or the GPU is
dedicated to TTS, you can omit --stage-overrides and raise
--gpu-memory-utilization (e.g. 0.3) for higher
throughput.
Predefined speaker voices (vivian, ryan,
etc.):
podman run --rm \
--device nvidia.com/gpu=all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-e TORCH_FLOAT32_MATMUL_PRECISION=high \
-e NVIDIA_TF32_OVERRIDE=1 \
-p 8091:8091 \
vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091 \
--gpu-memory-utilization 0.08 \
--stage-overrides '{"0":{"gpu_memory_utilization":0.08,"max_num_seqs":2},"1":{"gpu_memory_utilization":0.08,"max_num_seqs":2}}'Test:
curl -X POST http://0.0.0.0:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "Hello, how are you?", "voice": "vivian", "language": "English", "task_type": "CustomVoice"}' \
--output customvoice.wavDescribe the desired voice in natural language via the
instructions field:
podman run --rm \
--device nvidia.com/gpu=all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-e TORCH_FLOAT32_MATMUL_PRECISION=high \
-e NVIDIA_TF32_OVERRIDE=1 \
-p 8091:8091 \
vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --omni --port 8091 \
--gpu-memory-utilization 0.08 \
--stage-overrides '{"0":{"gpu_memory_utilization":0.08,"max_num_seqs":2},"1":{"gpu_memory_utilization":0.08,"max_num_seqs":2}}'Test:
curl -X POST http://0.0.0.0:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "Hello, how are you?", "voice": "test", "language": "English", "task_type": "VoiceDesign", "instructions": "A warm, cheerful young woman with a soft and expressive tone"}' \
--output voicedesign.wavClone a voice from a reference audio file. Mount the audio directory
into the container and pass --allowed-local-media-path:
podman run --rm \
--device nvidia.com/gpu=all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v /path/to/audio:/audio:ro \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-e TORCH_FLOAT32_MATMUL_PRECISION=high \
-e NVIDIA_TF32_OVERRIDE=1 \
-p 8091:8091 \
vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8091 \
--gpu-memory-utilization 0.08 \
--stage-overrides '{"0":{"gpu_memory_utilization":0.08,"max_num_seqs":2},"1":{"gpu_memory_utilization":0.08,"max_num_seqs":2}}' \
--allowed-local-media-path /audioTest:
curl -X POST http://0.0.0.0:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "Hello, how are you?", "voice": "cloned", "language": "English", "task_type": "Base", "ref_audio": "file:///audio/voice.wav", "ref_text": "Transcript of the reference audio goes here"}' \
--output baseclone.wavThe ref_text should contain only the spoken words from
the reference audio — omit non-speech sounds like laughs or pauses.
curl -X POST http://0.0.0.0:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "Hello, how are you?", "voice": "vivian", "language": "English"}' \
--output output.wavKey request fields:
input — text to synthesizevoice — speaker name (vivian,
ryan, etc.) for CustomVoice; natural language description
for VoiceDesignlanguage — Auto, English,
Chinese, etc.task_type — must match the loaded model:
CustomVoice, VoiceDesign, or
Baseresponse_format — wav (default),
mp3, flac, pcm, aac,
opusinstructions — style/emotion guidance; optional for
CustomVoice, required for VoiceDesign
(describes the voice), not applicable for Baseref_audio — reference audio path
(file:///audio/voice.wav) for Base modelref_text — transcript of the reference audio, used
alongside ref_audio for Base modelinitial_codec_chunk_frames — size of the first Code2Wav
decode block, in codec frames (12Hz codec). The server default favors
first-byte latency but then delivers audio slower than realtime for the
first seconds, forcing streaming clients to insert silence. With
8 (≈0.67s of audio in the first block) delivery never falls
behind realtime for +0.2s of first-byte latency; Open-LLM-VTuber’s
vllm_omni_tts engine sends this automatically on streaming
requestsThe /v1/audio/speech/stream endpoint accepts text
incrementally and emits PCM audio per sentence.
import asyncio, websockets, json, wave
async def main():
uri = "ws://0.0.0.0:8091/v1/audio/speech/stream"
async with websockets.connect(uri) as ws:
await ws.send(json.dumps({
"type": "session.config",
"voice": "Vivian",
"task_type": "CustomVoice",
"language": "Auto",
"stream_audio": True,
"response_format": "pcm",
"sample_rate": 24000,
}))
await ws.send(json.dumps({"type": "input.text", "text": "Hello! Streaming audio works."}))
await ws.send(json.dumps({"type": "input.done"}))
pcm_frames = b""
async for msg in ws:
if isinstance(msg, bytes):
pcm_frames += msg
else:
event = json.loads(msg)
if event.get("type") == "session.done":
break
with wave.open("output.wav", "wb") as wf:
wf.setnchannels(1)
wf.setsampwidth(2)
wf.setframerate(24000)
wf.writeframes(pcm_frames)
asyncio.run(main())Server events during streaming:
audio.start — sentence ready, metadataaudio.done — sentence completesession.done — all sentences processed