Skip to content

fix(agent): one embedder per PC, on the GPU, shown in Settings - #8368

Open
kovtcharov wants to merge 15 commits into
amd:mainfrom
kovtcharov:kalin/embedder-follows-profile-6008
Open

kovtcharov wants to merge 15 commits into
amd:mainfrom
kovtcharov:kalin/embedder-follows-profile-6008

Conversation

@kovtcharov

@kovtcharov kovtcharov commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Every chat on a PC now embeds documents and memory with one embedder, the GPU one, and Settings → Memory shows where it runs. Before, the embedder was chosen per session, so a chat on the other device could clear all of memory's vectors. And the NPU embedder that setup installed for NPU PCs (embed-gemma-300m-FLM) turned out to rank text no better than chance on FastFlowLM 1.0.7: on a Strix Halo, a paraphrase of a question scored below an unrelated sentence, and labelled retrieval on GAIA's own docs found the right passage about as often as picking at random. The GGUF build of the same model retrieves well. So NPU PCs now embed on the GPU, and the NPU option is shown but can't be chosen until a FastFlowLM release passes the new search check.

  • The setting is safe to change mid-conversation. A change reaches open chats on their next message and keeps their conversation and loaded skills. GAIA's agent is no longer restarted, so a reply in the terminal UI isn't cut off and memory still being saved isn't dropped.
  • Availability comes from the hardware, read the way setup reads it, not from the chat-model list (which has no embedders, so the check never refused anything).
  • A saved npu choice still loads, so no existing config stops starting; nothing offers or accepts it any more.
  • The NPU hardware test now checks search quality. It only checked vector shape before, which is how this passed. The NPU check is a strict expected failure, so a FastFlowLM release that fixes it fails the run and says to re-offer the NPU.

Not changed: the standalone email agent still picks the NPU embedder for its NPU triage model; it is being retired rather than fixed.

Refs #6008: fixes the embedder half of it; the empty NPU reply it also reports is a separate cause.

Test plan

  • python -m pytest tests/unit/test_memory_embedder_setting.py tests/unit/test_npu_flm_embedder.py tests/unit/test_memory_router.py tests/unit/chat/ui/test_upload_embedder_follows_profile.py tests/unit/test_init_command.py -q
  • python -m pytest hub/agents/gaia/python/tests/test_model_switch_follow.py hub/agents/gaia/python/tests/test_session_registry.py -q (an open chat follows a moved embedder and keeps its conversation and skills)
  • cd src/gaia/apps/webui && npx vitest run && npm run test:rtl -- -g "settings memory pane" (screenshots read in English and Arabic)
  • On a Strix Halo with Lemonade 2026.44.1 and FastFlowLM 1.0.7: python -m pytest tests/test_npu_flm_embedder_hw.py -k SearchQuality -rx passes the GPU check and xfails the NPU one
  • On GAIA's embedded Lemonade (2026.43.3, FastFlowLM 1.0.7): GAIA_REQUIRE_FLM=1 python -m pytest tests/test_npu_flm_embedder_hw.py -rx passes, with the NPU search check xfailing; alternating NPU chat turns and GPU embeds reloaded neither model

RAG and memory quality on NPU PCs changes with this; the GPU eval baseline isn't affected. Categories to watch on the next RC: rag_quality, gaia_rag, memory, gaia_memory.

…y chat

On a PC set up for the NPU, every chat still embedded documents and memory
with the GPU embedder, because the embedder came from a session's device
and the app never sets one. Each turn then loaded it on Vulkan next to the
FastFlowLM chat model and evicted it. The embedder now follows the PC's
profile (default_device): the FLM embedder on an NPU PC, the GGUF one
elsewhere, the same for every chat. Memory keeps one set of vectors from
one embedder, so a chat switched to the other device no longer clears them.

Closes amd#6008
@github-actions github-actions Bot added documentation Documentation changes tests Test changes agents labels Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Verdict: Request changes

On an NPU PC, chats now embed memory and documents with the NPU embedder instead of the GPU one, and the embedder no longer changes when a chat switches devices. That fixes the per-turn eviction for chat itself, and the one-embedder-per-PC reasoning holds up.

One path was missed, though: uploading a document in the Agent UI still indexes it with the GPU embedder on an NPU PC. That brings back the problem this PR fixes, because loading the GPU embedder evicts the NPU chat model. It also gets worse than before: the chat now searches with the NPU embedder, so the upload's vectors don't match and the whole document is embedded a second time on the first question. Indexing for an upload should take the same per-PC embedder.

Smaller point: issue #6008 also reports an empty chat reply on the NPU profile, which this PR doesn't touch. Use Refs #6008, or split that report out, so the issue doesn't close with half of it still open.

Real-world evidence

N/A: no evidence bundle was produced for this run, and the NPU behaviour needs a Ryzen AI PC. The verdict rests on static review alone. I couldn't run the PR's two tests here because pytest isn't installed on this runner. The test plan's on-hardware check (Lemonade's /health listing embed-gemma-300m-FLM with the FLM chat model still loaded) is still unchecked. It's worth running once the upload path is fixed, with a document attached.

🔍 Technical details

🟡 Agent UI upload indexing still uses the GGUF embedder (src/gaia/ui/_chat_helpers.py:6473)

_do_index builds RAGConfig(allowed_paths=allowed), so embedding_model falls back to DEFAULT_EMBEDDING_MODEL (src/gaia/rag/sdk.py:801, the GGUF model). On a PC with default_device: npu:

Before this PR, both paths used GGUF and at least agreed with each other. Found by grepping RAGConfig( outside the diff.

        from gaia.agents.registry import machine_embedding_model

        config = RAGConfig(
            allowed_paths=allowed, embedding_model=machine_embedding_model()
        )

_do_index also reuses batch["embedder"] across files, which is fine once every file uses the same model. A test like the new test_the_pcs_profile_picks_the_embedder, asserting the upload's RAGConfig.embedding_model on an npu profile, would pin this down.

🟢 Closes #6008 claims more than this PR fixes: the issue's first symptom (an empty reply containing Summarise user request) isn't addressed here.

Checked, no action needed

  • GaiaConfigError from machine_embedding_model() in _job_model is caught and re-raised as SetupPlanError by model_need (setup_plan.py:869-872), so it fails loudly as intended.
  • The new docs line ("memory is re-embedded") matches reconcile_embedder (it clears the vectors, then backfill re-embeds them). gaia init --profile npu sets default_device = "npu" (init_command.py:1963) and downloads embed-gemma-300m-FLM (init_command.py:255).
  • No other doc describes the embedder as following the session device.

Strengths

  • Pins the embedder to the machine rather than the model. That is the right invariant given that the memory store holds one embedder's vectors.
  • The flagship test covers both devices plus a switch_model, so the regression case is exercised end to end through build_session_agent.

The Agent UI and TUI run setup under the "gaia" profile on every PC, and
that set default_device to gpu. Now that the device picks the embedder,
running setup from the app on an NPU PC would switch it to the GPU
embedder and clear every memory vector. The "gaia" profile now keeps an
NPU PC on the NPU; only --profile npu or a GPU-only profile changes it.

The Memory dashboard also falls back to the PC's embedder instead of the
GGUF one, and its Rebuild uses the PC's embedder over a stamp left by an
earlier profile. A config that can't be read now says how to fix it.
@github-actions github-actions Bot added the cli CLI changes label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Verdict: Request changes

🟡 The earlier finding is still open. On an NPU PC, uploading a document in the Agent UI still indexes it with the GPU embedder. That load evicts the NPU chat model, which is the bug this PR fixes. Chat now searches with the NPU embedder, so the first question about the upload embeds the whole document a second time. Upload indexing should use the same per-PC embedder that chat uses. Add a test that sets the PC to the npu profile and checks which embedder the upload picks.

This push's setup change, which keeps an NPU PC on the NPU when the app runs setup, looks correct.

🔍 Technical details

src/gaia/ui/_chat_helpers.py:6473 still builds RAGConfig(allowed_paths=allowed), so it falls back to DEFAULT_EMBEDDING_MODEL (GGUF). _vectors_path hashes the embedder id, so the chat's FLM-backed RAG can't reuse the upload's vectors.

        from gaia.agents.registry import machine_embedding_model

        config = RAGConfig(
            allowed_paths=allowed, embedding_model=machine_embedding_model()
        )

The PR body still says Closes #6008. That issue's empty-reply symptom isn't fixed here, so Refs #6008 fits better.

On an NPU PC an upload still indexed with the GPU embedder, which loaded on
Vulkan and evicted the FLM chat model, and its vectors sat under a different
hash than chat's, so the first question re-embedded the whole document.
Uploads now use the same per-PC embedder every chat uses.
@kovtcharov-amd

Copy link
Copy Markdown
Collaborator

Uploads in the Agent UI now index with the same per-PC embedder that chat uses, so an NPU PC no longer loads the GPU embedder (and evicts the NPU chat model) when a document is added. The first question about an upload also no longer embeds it a second time. This closes the open review finding. The body now says Refs #6008, because the empty NPU reply in that issue is still open.

🔍 Technical details
  • _index_document builds RAGConfig(..., embedding_model=machine_embedding_model()). A corrupt config fails the upload with the actionable config error and does not fall back to GGUF.
  • New tests/unit/chat/ui/test_upload_embedder_follows_profile.py covers npu/gpu/cpu/no-profile/no-config, a whole folder in one embedder batch, and an unreadable config. 3 of its 7 cases fail without the fix.
  • Ran the affected unit suites (chat/ui, installer, init, memory router, NPU embedder, model-switch) on the PR head and on main: the same 7 Windows-local failures on both, none in this area.

…ser move it

Settings › Memory now names the embedding model and the device it runs on,
and offers the other device. The PC's setup still picks it by default (the
NPU after `gaia init --profile npu`); a choice the user makes is saved as
embedding_device, outlasts setup, and reaches chats already open, since
their agents and GAIA's agent are rebuilt with it. A device this PC can't
use is listed but can't be chosen, and a change waits for a running reply.
@kovtcharov kovtcharov changed the title fix(agent): embed on the NPU on an NPU PC, with one embedder for every chat fix(agent): one embedder per PC, on the NPU for an NPU PC, shown and movable in Settings Nov 16, 2026
kovtcharov and others added 6 commits November 16, 2026 15:59
The Agent UI shows this error (upload, embedder setting), and app text must
not send the user to a terminal. The wrapped config error already says to fix
or delete the file.
# Conflicts:
#	docs/guides/agent-ui.mdx
…s on their next turn

The Settings embedder choice offered the NPU on PCs without one: availability
was looked up in the chat-model list, which has no embedders, so it never
refused. It now asks the hardware the way setup does (an NPU present and set
up, a backend for the recipe).

A change no longer restarts GAIA's agent. That cut off replies from other
apps and dropped memory still being saved. Each open chat rebuilds on its
next turn instead, keeping its model, conversation and loaded skills, and the
old agent finishes its memory work before it closes. Choosing the embedder
already in use changes nothing.

Dashboard writes, Telegram uploads and the error text follow the PC's
embedder too.

Refs amd#6008
…com/kovtcharov/gaia into kalin/embedder-follows-profile-6008

# Conflicts:
#	src/gaia/agents/registry.py
FastFlowLM's NPU embedder (embed-gemma-300m-FLM, FLM 1.0.7) ranks text no
better than chance: a paraphrase scores below an unrelated sentence, and
labelled retrieval finds the right passage about as often as picking at
random. The GGUF build of the same model retrieves well, and its vectors
don't match the NPU build's, so the two can't share an index.

Every PC now embeds with the GPU embedder; `gaia init --profile npu`
downloads it. The NPU option is shown in Settings › Memory but can't be
chosen, and config refuses it, until a FastFlowLM release passes the new
search check. A choice saved earlier still loads. Lemonade keeps one model
per type, so the GPU embedder stays loaded beside the NPU chat model.

The NPU hardware test only checked vector shape, which is how this passed.
It now checks search quality: the GPU embedder must find each paraphrase,
and the NPU check is a strict xfail, so a fixed release fails the run.

Refs amd#6008
@kovtcharov kovtcharov changed the title fix(agent): one embedder per PC, on the NPU for an NPU PC, shown and movable in Settings fix(agent): one embedder per PC, on the GPU, shown in Settings Nov 17, 2026
@github-actions github-actions Bot added rag RAG system changes llm LLM backend changes performance Performance-critical changes labels Nov 17, 2026
Moving the embedder rebuilt an open chat's agent with its conversation and
loaded skills but dropped what the chat set on the old agent: an offline chat
went back online, a private one started saving to memory, and folders granted
mid-chat were lost. The rebuild now carries those over too, and the SPEC says
what a rebuild keeps.
…ents

The rebuild now also carries the attached and withheld document records,
read-only grants and withheld paths, so a document taken off on the same
turn as an embedder move loses its grant, and GAIA's own copy of a removed
document stays refused. Same carry-over as the MCP settings rebuild.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents cli CLI changes documentation Documentation changes llm LLM backend changes performance Performance-critical changes rag RAG system changes tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants