A Voice Assistant That Could Not See Color

This Post is the build log of a voice assistant for Home Assistant that sends nothing to a cloud. It has the container config, the errors, the numbers for four models on two GPUs, and an audit of the tool calls behind a wrong answer.

The setup: Home Assistant 2026.7 on a Raspberry Pi 4. A Proxmox host with a VM that holds a GeForce GTX 980 (4 GB). A gaming PC with an RTX 4080 (16 GB) and Ollama.

The pipeline

A Home Assistant voice pipeline has three stages, and each one is a network service that you can place on any machine:

1. Speech to text. Whisper, over the Wyoming protocol.
2. A conversation agent. The built-in intent engine, or a language model.
3. Text to speech. Piper, over the Wyoming protocol.

Text to speech: Piper

Piper runs on the CPU and is not the problem in any setup.

piper:
  image: rhasspy/wyoming-piper:latest
  command: --voice en_US-lessac-medium
  ports: ["10200:10200"]
  volumes: ["piper:/data"]

It makes 3 seconds of audio in 0.2 seconds on six cores of a Ryzen 2700X.

Speech to text: three errors on a GTX 980

The plan was Whisper on the GPU, with the linuxserver/faster-whisper image. That image runs faster-whisper on CTranslate2. A GTX 980 is a Maxwell card with CUDA compute capability 5.2, and it failed in three steps.

The gpu image with its defaults:

ValueError: Requested float16 compute type, but the target device or
backend do not support efficient float16 computation.

With int8:

ValueError: Requested int8 compute type, but the target device or
backend do not support efficient int8 computation.

CTranslate2 can tell you what a device supports:

>>> ctranslate2.get_supported_compute_types("cuda")
{'float32'}

With float32 the model loads, and nvidia-smi shows 1,849 MiB in use. Then each request fails:

RuntimeError: parallel_for failed: cudaErrorInvalidDevice: invalid device ordinal

The model is in memory, but the compiled kernels have no code for this architecture. The gpu-legacy tag of the image, which exists for older cards, gives the same error. So Whisper runs on the CPU:

whisper:
  image: lscr.io/linuxserver/faster-whisper:latest
  ports: ["10300:10300"]
  environment:
    WHISPER_MODEL: small-int8
    WHISPER_LANG: en
    WHISPER_BEAM: "1"

small-int8 on six cores turns a command of 3 seconds into text in about 2.6 seconds.

The same card runs Ollama without a problem, because Ollama still ships a CUDA 12 build that accepts compute capability 5.2. Support for an old GPU is a property of each library, not of the card.

A test with no microphone

Wyoming is a simple protocol: one JSON header line for each event, then an optional payload. That makes a round trip test easy. Piper speaks a sentence, and Whisper writes down what it hears.

def send(sock, type_, data=None, payload=b""):
    data_bytes = json.dumps(data or {}).encode()
    header = {"type": type_, "data_length": len(data_bytes),
              "payload_length": len(payload)}
    sock.sendall(json.dumps(header).encode() + b"\n" + data_bytes + payload)

# Piper: send "synthesize", collect the "audio-chunk" payloads until "audio-stop".
# Whisper: send "transcribe", "audio-start", the chunks, "audio-stop", read "transcript".

said:    Turn on the office lights and set the kitchen to fifty percent.
heard:   Turn on the office lights and set the kitchen to 50%.
piper:   0.20 s for 3.0 s of audio
whisper: 2.72 s

Each test is the same as the last one, which a human voice is not.

Wiring it into Home Assistant by API

All of it can be done without the UI, which matters when an agent does the work.

The Wyoming integrations go in through the config flow endpoint. One call starts the flow, a second call answers the form:

POST /api/config/config_entries/flow            {"handler": "wyoming"}
POST /api/config/config_entries/flow/<flow_id>  {"host": "192.168.50.30", "port": 10300}

The pipeline is a websocket command:

{"type": "assist_pipeline/pipeline/create", "name": "Nyx (local)", "language": "en",
 "conversation_engine": "conversation.home_assistant", "conversation_language": "en",
 "stt_engine": "stt.faster_whisper", "stt_language": "en",
 "tts_engine": "tts.piper", "tts_language": "en_US", "tts_voice": "en_US-lessac-medium",
 "wake_word_entity": null, "wake_word_id": null, "prefer_local_intents": true}

A full pipeline run with audio also works over the websocket. assist_pipeline/run with start_stage: "stt" answers with a run-start event that holds stt_binary_handler_id. Each audio chunk then goes as a binary frame with that id as its first byte, and one frame with only the id ends the audio. The audio must be 16 kHz, 16-bit, mono. Piper gives 22,050 Hz, so the test resamples it.

One trap on the way: I stored the Home Assistant token in the macOS keychain with security add-generic-password -w and pasted it at the prompt. That prompt keeps 128 characters. The token has 183. Every call answered 401 until I compared the lengths. -w "$(pbpaste)" on the command line does not truncate.

One sentence, one action

"Turn off the office lights" worked on the first try. "Turn on the office lights and make them green" did not. The built-in agent matches a sentence against patterns, and one pattern gives one action.

The fix is a language model as the conversation agent. Home Assistant gives it the sentence, the exposed entities, and a set of tools. With prefer_local_intents on, a sentence that matches a pattern never reaches the model.

The cost is in the prompt. With 60 exposed entities, each request carried 2,000 to 3,400 tokens before the first output token.

Four models, two GPUs

All through the Ollama integration, with num_ctx at 8192. The test sentence: "turn on the office lights and make them green".

The numbers behind the first two rows:

• llama3.2:3b spent 5.2 seconds on prompt evaluation alone: 2,022 tokens at 387 tokens each second. It fits the card and it cannot use the tools.
• qwen3:4b uses the tools correctly. With an 8k context it needs 4.37 GB, so about a third of it runs on the CPU, at 13 tokens each second. It also ignored the request to turn off thinking and wrote its reasoning into the reply.

The third row is the same lesson on new hardware. A 36B model on a 16 GB card is slower than a 9B model that fits. qwen3.5:9b runs at 95 tokens each second:

So the pipeline is split across machines. Speech runs on the server that is always on. The model runs on the fast card. End to end, the reply starts about 4.5 seconds after the end of speech: 2.6 for Whisper, 1.5 for the model, and the rest for Piper and the network.

The audit

By voice, the results were mixed. Home Assistant keeps a debug record of each pipeline run, and it is readable over the websocket:

{"type": "assist_pipeline/pipeline_debug/list", "pipeline_id": "..."}
{"type": "assist_pipeline/pipeline_debug/get", "pipeline_id": "...", "pipeline_run_id": "..."}

The intent-progress events hold each tool call and each tool result. Here are five runs in a row.

After the third run, the assistant said this:

The office lights have been reset to normal brightness (100%). The color has been removed, so they're back to their default white light.

The lights were bright blue. The tool call explains the claim:

{"tool_name": "HassLightSet", "tool_args": {"name": "Office", "brightness": 100}}

It set the brightness and said that the color was gone. On my complaint, it read the state of the light. This is the full tool result:

Live Context: An overview of the areas and the devices in this smart home:
- names: Office
  domain: light
  state: 'on'
  areas: Office
  attributes:
    brightness: '255'

No color. No color temperature. The model cannot see the one fact I was talking about. It then made the same call again and suggested that I check my device settings.

Three causes, and only one is about the model:

• The live context has no color attribute. The model was blind to the state in question.
• "Change it back" has no target. No record of the earlier state exists in the context.
• The model stated a result that it did not check. That one is its own.

The fix: a prompt, not a model

I added rules to the instructions of the agent:

Rules for lights:
- You cannot see the color or the color temperature of a light. The live context
  shows only on or off and the brightness. Never say what color a light has now,
  unless your own tool call in this conversation set it.
- Brightness does not change color. To remove a color, set a color temperature.
- The normal state of a light in this home is warm white, 2700 K, at 100% brightness.
  When the user says "change it back", "reset", "undo", "normal", or "white", set the
  temperature to 2700 and the brightness to 100 in one tool call.
- When the user says a change did not work, do not repeat the same tool call. Make a
  different call that sets the missing value.

Rules for answers:
- Say only what your tool calls did, with the values you sent. Do not say that a
  device has a fault.
- If you did not call a tool, do not say that something changed.

The same sentences again, in one conversation, with the real state of the light after each:

In a new conversation I asked what color the lights were. It answered that it cannot see color, only on or off and brightness.

What I take from this

• Check the compute types before you plan around an old GPU. ctranslate2.get_supported_compute_types("cuda") would have saved an hour.
• A model that fits beats a larger model that does not. This held on a 4 GB card and on a 16 GB card.
• Count the prompt. The entity list was 2,000 to 3,400 tokens for each sentence. Fewer exposed entities is the cheapest speedup.
• Read the tool calls, not the answers. The answer said the color was removed. The call said brightness only.
• Tell the model what it cannot see. The invented answers stopped when the prompt named the gap in the tool result.
• Speech to text is the weak link now. It is more than half of the latency, and it invents words from short audio. The fast card could do that job too.