vLLM-Omni (TTS)

Overview

vLLM-Omni extends vLLM with omni-modality inference (audio, video, image). It runs as a Podman container, serving an OpenAI-compatible TTS API on port 8091.

The Qwen3-TTS family has 5 model variants across 3 task types. Each instance serves exactly one model — to run multiple simultaneously, use separate containers on different ports.

Model Task Size
Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice Predefined speaker voices 1.7B
Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice Predefined speaker voices 0.6B
Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign Describe voice in natural language 1.7B
Qwen/Qwen3-TTS-12Hz-0.6B-VoiceDesign Describe voice in natural language 0.6B
Qwen/Qwen3-TTS-12Hz-1.7B-Base Voice cloning from reference audio 1.7B

The task_type field in API requests must match the loaded model (CustomVoice, VoiceDesign, or Base).

Setup

Create a Containerfile that layers vllm-omni on top of the official vLLM image:

FROM vllm/vllm-openai:v0.20.0
RUN pip install vllm-omni==0.20.0rc1
RUN pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.4/flash_attn-2.8.3+cu130torch2.11-cp312-cp312-linux_x86_64.whl"
ENTRYPOINT ["vllm-omni"]

The flash-attn line installs Flash Attention 2 from a prebuilt wheel — no compilation needed. The wheel must match the CUDA, PyTorch, and Python versions inside the base image. To find yours:

  1. Query the base image versions:

    podman run --rm --entrypoint python3 vllm/vllm-openai:v0.20.0 \
      -c "import torch, sys; print(f'CUDA: {torch.version.cuda}  Torch: {torch.__version__}  Python: {sys.version.split()[0]}')"
  2. Browse the mjun0812/flash-attention-prebuild-wheels releases (wheels built via GitHub Actions from source). Find the .whl file matching your cu{CUDA}torch{TORCH}-cp{PYTHON}-cp{PYTHON}-linux_x86_64 pattern and replace the URL above.

Build the image:

podman build -t vllm-omni .

Running the Server

Each instance serves exactly one model — stop one before starting another on the same port. The task_type in API requests must match the loaded model.

The -v ~/.cache/huggingface mount reuses already-downloaded model weights across container runs. The API is available at http://0.0.0.0:8091 (not localhost — Podman networking quirk).

Environment variables

All run commands below include these performance flags:

Variable Purpose
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True Reduces CUDA allocator fragmentation, helping Code2Wav graph capture succeed when free VRAM is tight
TORCH_FLOAT32_MATMUL_PRECISION=high Enables TF32 matmuls in PyTorch on Ampere+ GPUs (~10–20% faster)
NVIDIA_TF32_OVERRIDE=1 Forces TF32 at the CUDA driver level (belt-and-suspenders with the above)

Memory tuning

vLLM-Omni’s 2-stage architecture (Talker + Code2Wav) spawns two independent engines. By default each pre-allocates a large KV cache pool, which wastes VRAM on TTS workloads.

For a 12 GB GPU sharing VRAM with an LLM (e.g. Ollama), the 1.7B model fits in ~8 GB with these flags, leaving room for a ~2–3 GB LLM. Without them, vLLM defaults to gpu_memory_utilization=0.9 and will consume nearly all VRAM.

Given that you have 24+ GB of VRAM or the GPU is dedicated to TTS, you can omit --stage-overrides and raise --gpu-memory-utilization (e.g. 0.3) for higher throughput.

CustomVoice

Predefined speaker voices (vivian, ryan, etc.):

podman run --rm \
  --device nvidia.com/gpu=all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e TORCH_FLOAT32_MATMUL_PRECISION=high \
  -e NVIDIA_TF32_OVERRIDE=1 \
  -p 8091:8091 \
  vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091 \
  --gpu-memory-utilization 0.08 \
  --stage-overrides '{"0":{"gpu_memory_utilization":0.08,"max_num_seqs":2},"1":{"gpu_memory_utilization":0.08,"max_num_seqs":2}}'

Test:

curl -X POST http://0.0.0.0:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you?", "voice": "vivian", "language": "English", "task_type": "CustomVoice"}' \
  --output customvoice.wav

VoiceDesign

Describe the desired voice in natural language via the instructions field:

podman run --rm \
  --device nvidia.com/gpu=all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e TORCH_FLOAT32_MATMUL_PRECISION=high \
  -e NVIDIA_TF32_OVERRIDE=1 \
  -p 8091:8091 \
  vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --omni --port 8091 \
  --gpu-memory-utilization 0.08 \
  --stage-overrides '{"0":{"gpu_memory_utilization":0.08,"max_num_seqs":2},"1":{"gpu_memory_utilization":0.08,"max_num_seqs":2}}'

Test:

curl -X POST http://0.0.0.0:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you?", "voice": "test", "language": "English", "task_type": "VoiceDesign", "instructions": "A warm, cheerful young woman with a soft and expressive tone"}' \
  --output voicedesign.wav

Base (voice cloning)

Clone a voice from a reference audio file. Mount the audio directory into the container and pass --allowed-local-media-path:

podman run --rm \
  --device nvidia.com/gpu=all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /path/to/audio:/audio:ro \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e TORCH_FLOAT32_MATMUL_PRECISION=high \
  -e NVIDIA_TF32_OVERRIDE=1 \
  -p 8091:8091 \
  vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8091 \
  --gpu-memory-utilization 0.08 \
  --stage-overrides '{"0":{"gpu_memory_utilization":0.08,"max_num_seqs":2},"1":{"gpu_memory_utilization":0.08,"max_num_seqs":2}}' \
  --allowed-local-media-path /audio

Test:

curl -X POST http://0.0.0.0:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you?", "voice": "cloned", "language": "English", "task_type": "Base", "ref_audio": "file:///audio/voice.wav", "ref_text": "Transcript of the reference audio goes here"}' \
  --output baseclone.wav

The ref_text should contain only the spoken words from the reference audio — omit non-speech sounds like laughs or pauses.

API Usage

Speech synthesis

curl -X POST http://0.0.0.0:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you?", "voice": "vivian", "language": "English"}' \
  --output output.wav

Key request fields:

Streaming (WebSocket)

The /v1/audio/speech/stream endpoint accepts text incrementally and emits PCM audio per sentence.

import asyncio, websockets, json, wave

async def main():
    uri = "ws://0.0.0.0:8091/v1/audio/speech/stream"
    async with websockets.connect(uri) as ws:
        await ws.send(json.dumps({
            "type": "session.config",
            "voice": "Vivian",
            "task_type": "CustomVoice",
            "language": "Auto",
            "stream_audio": True,
            "response_format": "pcm",
            "sample_rate": 24000,
        }))
        await ws.send(json.dumps({"type": "input.text", "text": "Hello! Streaming audio works."}))
        await ws.send(json.dumps({"type": "input.done"}))

        pcm_frames = b""
        async for msg in ws:
            if isinstance(msg, bytes):
                pcm_frames += msg
            else:
                event = json.loads(msg)
                if event.get("type") == "session.done":
                    break

        with wave.open("output.wav", "wb") as wf:
            wf.setnchannels(1)
            wf.setsampwidth(2)
            wf.setframerate(24000)
            wf.writeframes(pcm_frames)

asyncio.run(main())

Server events during streaming:

References