Fix role-label leakage and add a local Supermemory server

README.md

@@ -22,7 +22,27 @@ npm install @supermemory/eve
 ```
 
 The provider supports Eve 0.47.3 and newer 0.x releases. Set `SUPERMEMORY_API_KEY` from the
-[Supermemory console](https://console.supermemory.ai).
+[Supermemory console](https://console.supermemory.ai). For a self-hosted server, set
+`SUPERMEMORY_BASE_URL` (for example `http://localhost:6767`). Both are read when the client is
+first used. `SUPERMEMORY_LOG` sets the SDK log level.
+
+## Local server
+
+```bash
+npm run supermemory:up
+```
+
+That builds official `supermemory-server` 0.0.8, starts it on `http://127.0.0.1:6767`, and writes
+`SUPERMEMORY_API_KEY` and `SUPERMEMORY_BASE_URL` to gitignored `.env.local`. `npm run supermemory:down`
+stops the container and keeps its data volume.
+
+Put provider settings in `.env` or `.env.local`. A custom `OPENAI_BASE_URL` makes the server
+call `{base}/chat/completions`. For Azure AI Foundry, set that base URL to
+`https://<resource>.services.ai.azure.com/openai/v1` and `OPENAI_MODEL` to the deployment name.
+Foundry chat models reject `max_tokens`, so an Azure base URL also starts a small rewrite proxy.
+A trailing `/responses` on the base URL is removed. Embeddings stay on the local model. For
+Ollama on this machine, set `OPENAI_BASE_URL=http://host.docker.internal:11434/v1` and any
+non-empty `OPENAI_API_KEY`.
 
 Create a memory slot in the consuming Eve agent. `supermemory(...)` configures the provider;
 `defineMemory(...)` binds it to an Eve-managed scope.
@@ -153,6 +173,22 @@ export default defineMemory({
 need a different memory policy. The model cannot change caller identity, container routing, or
 capture policy at runtime.
 
+`client` is passed to the Supermemory SDK constructor. Leave a field out and the SDK keeps its
+default, including `SUPERMEMORY_BASE_URL` and `SUPERMEMORY_LOG`. Set `client.baseURL` when the
+URL should come from code instead of the environment.
+
+```ts
+provider: supermemory({
+  client: {
+    baseURL: "http://localhost:6767",
+    timeout: 20_000,
+  },
+}),
+```
+
+`supermemory()` with no options is enough when `SUPERMEMORY_API_KEY` is set in the environment
+that runs the agent.
+
 ## Frequently asked questions
 
 ### How do I add memory to Eve?
@@ -183,8 +219,11 @@ Requires Node.js 24 or newer.
 
 ```bash
 npm install
+npm test
 npm run check
 npm run typecheck
 npm run build
 npm pack --dry-run
 ```
+
+Live agent evals live in `evals/` and run with `eve eval` from a consuming Eve app. See `evals/README.md`.

biome.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://biomejs.dev/schemas/2.5.7/schema.json",
   "files": {
-    "includes": ["src/**/*.ts"]
+    "includes": ["src/**/*.ts", "evals/**/*.ts", "tests/**/*.ts"]
   },
   "formatter": {
     "enabled": true,

docker-compose.yml

@@ -0,0 +1,36 @@
+services:
+  # Foundry chat deployments reject max_tokens. This rewrites that field and
+  # forwards to OPENAI_BASE_URL. Started only with the azure compose profile.
+  azure-shim:
+    profiles: ["azure"]
+    image: node:24-bookworm-slim
+    command: ["node", "/shim/azure-chat-shim.mjs"]
+    environment:
+      PORT: "8080"
+      AZURE_UPSTREAM: ${OPENAI_BASE_URL}
+    volumes:
+      - ./scripts/azure-chat-shim.mjs:/shim/azure-chat-shim.mjs:ro
+
+  supermemory:
+    build:
+      context: ./docker
+    image: eve-supermemory-local:0.0.8
+    ports:
+      - "127.0.0.1:6767:6767"
+    environment:
+      PORT: "6767"
+      SUPERMEMORY_DATA_DIR: /data
+      SUPERMEMORY_EMBEDDING_PROVIDER: local
+      SUPERMEMORY_DISABLE_TELEMETRY: "1"
+    env_file:
+      - path: .env
+        required: false
+      - path: .env.local
+        required: false
+      - path: .env.supermemory
+        required: false
+    volumes:
+      - supermemory-data:/data
+
+volumes:
+  supermemory-data:

docker/Dockerfile

@@ -0,0 +1,40 @@
+# Official self-hosted server. There is no upstream image; the release is one binary.
+# https://github.com/supermemoryai/supermemory/releases/tag/server-v0.0.8
+FROM debian:bookworm-slim
+
+ARG SUPERMEMORY_VERSION=0.0.8
+ARG TARGETARCH
+
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends ca-certificates curl \
+  && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /opt/supermemory
+
+RUN set -eu; \
+  case "$TARGETARCH" in \
+    amd64) PLATFORM=linux-x64 ;; \
+    arm64) PLATFORM=linux-arm64 ;; \
+    *) echo "unsupported architecture: $TARGETARCH" >&2; exit 1 ;; \
+  esac; \
+  BASE="https://github.com/supermemoryai/supermemory/releases/download/server-v${SUPERMEMORY_VERSION}"; \
+  curl -fsSL "$BASE/manifest.json" -o /tmp/manifest.json; \
+  curl -fsSL "$BASE/supermemory-server-${PLATFORM}" -o /opt/supermemory/supermemory-server; \
+  EXPECTED="$(grep -A2 "\"${PLATFORM}\"" /tmp/manifest.json | grep -oE '[a-f0-9]{64}' | head -n 1)"; \
+  test -n "$EXPECTED"; \
+  echo "${EXPECTED}  /opt/supermemory/supermemory-server" | sha256sum -c -; \
+  chmod 755 /opt/supermemory/supermemory-server; \
+  rm /tmp/manifest.json
+
+ENV PORT=6767 \
+  SUPERMEMORY_DATA_DIR=/data \
+  SUPERMEMORY_EMBEDDING_PROVIDER=local \
+  SUPERMEMORY_DISABLE_TELEMETRY=1
+
+EXPOSE 6767
+VOLUME ["/data"]
+
+HEALTHCHECK --interval=5s --timeout=3s --start-period=120s --retries=30 \
+  CMD curl -sS -o /dev/null http://127.0.0.1:6767/ || exit 1
+
+ENTRYPOINT ["/opt/supermemory/supermemory-server"]

evals/README.md

@@ -0,0 +1,38 @@
+# Live Eve evals
+
+These evals run an Eve agent against Supermemory. Each run uses its own container prefix. Upstream pull request #1 targeted the old extension, so these cases now follow the memory provider.
+
+`eve eval` discovers `evals/*.eval.ts` next to `agent/`. Add that agent in this repository. The helpers import this package's source, and they derive the container tag from Eve's scope key.
+
+```ts
+// agent/memory/supermemory.ts
+import supermemory from "@supermemory/eve";
+import { defineMemory } from "eve/memory";
+
+export default defineMemory({
+  namespace: process.env.SUPERMEMORY_E2E_NAMESPACE,
+  description: "Recall and manage durable context for the current user.",
+  provider: supermemory({
+    containerTagPrefix: process.env.SUPERMEMORY_E2E_PREFIX,
+  }),
+  scope: process.env.SUPERMEMORY_E2E_SCOPE ?? "e2e",
+});
+```
+
+Also export `askQuestion()` from `agent/tools/ask_question.ts`. The slot file name must stay `supermemory.ts` so the tools are `supermemory__*`.
+
+```text
+OPENAI_API_KEY
+SUPERMEMORY_API_KEY
+SUPERMEMORY_E2E_PREFIX
+SUPERMEMORY_E2E_NAMESPACE
+SUPERMEMORY_E2E_RUN_ID
+```
+
+`SUPERMEMORY_E2E_SCOPE` defaults to `e2e`. `SUPERMEMORY_BASE_URL` points the helpers at a self-hosted server. Omit it for `https://api.supermemory.ai`.
+
+`npm run supermemory:up` writes `SUPERMEMORY_API_KEY` and `SUPERMEMORY_BASE_URL=http://127.0.0.1:6767` into `.env.local`. The server and the Eve agent each need a model key. One `OPENAI_API_KEY` covers both.
+
+```bash
+eve eval --max-concurrency 1 --timeout 180000 --verbose
+```

evals/capture.eval.ts

@@ -0,0 +1,30 @@
+import { defineEval } from "eve/evals";
+import { equals, includes } from "eve/evals/expect";
+
+import { waitForDocumentContent } from "./helpers.js";
+
+export default defineEval({
+  async test(t) {
+    const update = "The staging deployment completed successfully.";
+    const turn = await t.send(`Acknowledge this update in one short sentence: ${update}`);
+    turn.expectOk();
+    turn.usedNoTools();
+
+    const customId = `conv_${turn.sessionId}`;
+    const document = await waitForDocumentContent(customId, update);
+    const metadata = document?.metadata;
+
+    t.check(document?.customId, equals(customId));
+    t.check(document?.content, includes(update));
+    t.check(document?.content, includes("[user]"));
+    t.check(
+      metadata && typeof metadata === "object" ? Reflect.get(metadata, "source_type") : null,
+      equals("conversation"),
+    );
+    t.check(
+      metadata && typeof metadata === "object" ? Reflect.get(metadata, "session_id") : null,
+      equals(turn.sessionId),
+    );
+    t.succeeded();
+  },
+});

evals/evals.config.ts

@@ -0,0 +1,3 @@
+import { defineEvalConfig } from "eve/evals";
+
+export default defineEvalConfig({});

evals/extract-file.eval.ts

@@ -0,0 +1,29 @@
+import { dirname, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { defineEval } from "eve/evals";
+import { includes } from "eve/evals/expect";
+
+export default defineEval({
+  async test(t) {
+    const fixturePath = resolve(
+      dirname(fileURLToPath(import.meta.url)),
+      "fixtures/extraction-source.txt",
+    );
+    const session = await t.session();
+    const turn = await session.sendFile(
+      "Add this migration brief to Supermemory so we can search it in later sessions. Once it is indexed, read it back and confirm when the production freeze begins and who owns the rollback.",
+      fixturePath,
+      "text/plain",
+    );
+
+    turn.expectOk();
+    turn.calledTool("supermemory__extract", { input: { kind: "file" } });
+    turn.calledTool("supermemory__read_document", {
+      input: { container: "agent_extraction" },
+    });
+    t.check(turn.message, includes("October 14"));
+    t.check(turn.message, includes("Priya Nair"));
+    t.succeeded();
+  },
+});

evals/fixtures/extraction-source.txt

@@ -0,0 +1,9 @@
+Northstar vendor migration brief
+
+The production migration freeze begins on October 14 at 6:00 PM Pacific. During
+the freeze, the support team should route urgent customer-impacting changes to
+the incident channel instead of deploying them directly.
+
+Priya Nair owns the rollback decision. The database team should preserve the
+pre-migration snapshot until Priya confirms that the first integrity check has
+passed.

evals/forget-exact.eval.ts

@@ -0,0 +1,29 @@
+import { defineEval } from "eve/evals";
+
+import { createMemories, waitForForgottenMemory } from "./helpers.js";
+
+export default defineEval({
+  async test(t) {
+    const fact = "daily standups at 8:30 AM";
+    const created = await createMemories([
+      {
+        content: `The current user prefers ${fact}.`,
+        metadata: { source: "user_preference" },
+      },
+    ]);
+    const memory = created.memories[0];
+    if (!memory) throw new Error("Direct memory creation returned no memory.");
+
+    const turn = await t.send(
+      `I no longer want you to remember that I prefer ${fact}. Forget that one preference.`,
+    );
+    turn.expectOk();
+    turn.calledTool("supermemory__search", { input: { scope: "memories" } });
+    turn.calledTool("supermemory__forget", {
+      input: { memoryId: memory.id },
+      count: 1,
+    });
+    await waitForForgottenMemory(memory.id);
+    t.succeeded();
+  },
+});

evals/forget-matching.eval.ts

@@ -0,0 +1,52 @@
+import { defineEval } from "eve/evals";
+
+import { createMemories, waitForForgottenMemory } from "./helpers.js";
+
+export default defineEval({
+  async test(t) {
+    const topic = "Juniper relocation";
+    const created = await createMemories([
+      {
+        content: `The current user plans to relocate to Portland in October as part of the ${topic}.`,
+        metadata: { source: "relocation_planning" },
+      },
+      {
+        content: `The current user prefers morning apartment tours for the ${topic}.`,
+        metadata: { source: "relocation_planning" },
+      },
+    ]);
+    const ids = created.memories.map((memory) => memory.id);
+    if (ids.length !== 2) throw new Error("Direct memory creation did not return two memories.");
+
+    const preview = await t.send(
+      `I'm no longer planning the ${topic}. Forget everything you remember about that relocation, but show me what would be removed before deleting anything.`,
+    );
+    preview.expectOk();
+    preview.calledTool("supermemory__forget_matching", {
+      input: { dryRun: true },
+      count: 1,
+    });
+
+    const request = preview.session.requireInputRequest({ toolName: "ask_question" });
+    const confirm = request.options?.find((option) =>
+      /remove|forget|yes|confirm|approve/i.test(`${option.label} ${option.id}`),
+    );
+    if (!confirm) {
+      throw new Error("ask_question did not offer a confirm option.");
+    }
+
+    const finalized = await preview.session.respond([
+      { requestId: request.requestId, optionId: confirm.id },
+    ]);
+    finalized.expectOk();
+    finalized.calledTool("supermemory__forget_matching", {
+      input: {
+        dryRun: false,
+        ids: (value) => Array.isArray(value) && ids.every((id) => value.includes(id)),
+      },
+      count: 1,
+    });
+    await Promise.all(ids.map((id) => waitForForgottenMemory(id)));
+    t.succeeded();
+  },
+});