Haven provider + fix dropped completions on vLLM backends - #1
Open
coelhogonzalo wants to merge 4 commits into
Open
Haven provider + fix dropped completions on vLLM backends#1coelhogonzalo wants to merge 4 commits into
coelhogonzalo wants to merge 4 commits into
Conversation
…roxy) Adds provider: haven backed by the haven-proxy package's relay core: request bodies are HPKE-encrypted end-to-end to the Tinfoil enclave on this machine, so Haven never sees prompts or completions and no localhost proxy process is needed. - core/llm/llms/Haven.ts: OpenAI subclass on the raw-fetch path (useOpenAIAdapterFor = []); POST chat/completions goes through createSecureRelay, everything else through the normal fetch with key-free headers (the relay injects X-Api-Key itself, only to the Haven origin). Key resolution: config apiKey -> HAVEN_API_KEY -> ~/.haven-proxy/config.json, so the key never has to live in a shareable config file. - Registration, tool-support (per-model tool_call flags from the Haven catalog) and chat-templating entries. - config_schema.json: provider enum entry, description, model enum. - haven-proxy.d.ts: ambient types (the package ships none). - Haven.vitest.ts: relay-mocked coverage of streaming, buffered-SSE synthesis, non-stream JSON, error mapping, abort, key resolution, header hygiene, endpoint shape and context lengths.
gui and binary compile core sources with their own tsconfigs, which do not include core's .d.ts files, so the haven-proxy module declarations were out of scope there and the gui build failed. A triple-slash reference in Haven.ts pulls the declarations into every program that compiles the file.
chatCompletionStream deferred any chunk with a `usage` field to the end of the stream, assuming OpenAI's convention where usage arrives once on a final chunk with no choices. vLLM attaches usage to every chunk, so each one was stored into lastChunkWithUsage and never yielded; only the final chunk survived, and its delta is empty. The caller assembled an empty completion with no error, so the request was logged as a success with 0 generated tokens and the UI showed no reply at all. This hit any vLLM-backed provider: openai endpoint. Defer a chunk only when it has no choices, so content chunks pass through with their usage attached and OpenAI's usage-only chunk is still emitted last. Add tests covering all three stream shapes.
Distinguishes local builds carrying the Haven provider and the streaming fix from the 1.3.40 marketplace release, so an installed VSIX is identifiable and VS Code does not treat the marketplace build as an equal-version replacement. Also records the haven-proxy dependency in the gui lockfile, which the Haven provider commit added to core.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opened against this fork's
mainto triggerpr-build-upload-vsix.yaml, whichbuilds the VSIX natively for linux-x64, win32-x64 and darwin-arm64 and uploads
each as an artifact. The darwin-arm64 artifact is the one to share with macOS
users. Not intended for
continuedev/continue.Commits
provider: haven, an OpenAI subclass onthe raw-fetch path. Request bodies are HPKE-encrypted in-process to the
Tinfoil enclave via haven-proxy's relay core, so no localhost proxy runs and
Haven never sees prompts or completions.
The streaming fix
chatCompletionStreamdeferred any chunk carrying ausagefield to the end ofthe stream, assuming OpenAI's convention where usage arrives once on a final
chunk with no choices. vLLM attaches usage to every chunk — measured 59 of
59 frames against a live endpoint — so each chunk was stored into
lastChunkWithUsageand never yielded. Only the final chunk survived, and itsdelta is empty.
The caller assembled an empty completion without raising, so the request logged
as a success with 0 generated tokens and the UI showed no reply at all. This
affected any vLLM-backed
provider: openaiendpoint, not just Haven.The fix defers a chunk only when it has no choices, so content chunks pass
through with their usage attached while OpenAI's usage-only chunk is still
emitted last. Three tests cover all three stream shapes; reverting the fix fails
exactly the vLLM one with
expected '' to be 'Hi there'.This commit is independent of the Haven work and cherry-picks cleanly onto
mainif it is worth sending upstream.Verification
packages/openai-adapters: 3/3 new tests pass,tsc --noEmitclean.core/llm/llms/Haven.vitest.ts: 10/10 pass.in the shipped bundle and that chat returns non-zero generated tokens.
🤖 Generated with Claude Code