Plugin SDK proposal: exact totals and resumable read-only query pages for mail, contacts, calendars and files

[yontani]

Proposal

Would a small, read-only query-page contract with exact totals and stable continuation fit the plugin SDK? I am building a plugin that must handle mailboxes, contacts, calendars and stored files well beyond one result page, including requests such as “how many messages match this filter?” and resumable bulk reads.

An implementation example is merged in my fork:

• Fork PR #2: bounded read-only plugin query pages
• Contract, examples and limitations at the merged revision
• JMAP page implementation
• Validated sandbox boundary

This is a proposal for feedback on the API shape. Existing email.search, email.list, contacts.list and expanded calendar.getEvents remain unchanged in the example.

Why a separate page envelope?

The existing APIs are useful for interactive plugin reads, but they do not expose the same completeness contract:

• email.search returns a bounded newest-first array without a continuation or exact total.
• email.list accepts a position but returns only summaries, discarding the query metadata.
• Contact listing hides the client's internal enumeration behind a materialized array.
• calendar.getEvents returns expanded occurrences in a bounded range, capped at 1,000; a stored-record count is a different question.
• A plugin should not need a raw JMAP request primitive or credentials to implement these workflows for files.

Increasing a result cap is not enough: consumers need to know whether they reached the end, whether an exact total is available, and whether the result set changed while paging.

Example surface

API	Existing permission	Counted unit
api.email.query(options?)	email:read	messages
api.contacts.query(options?)	contacts:read	contact cards
api.calendar.queryEvents(options?)	calendar:read	stored event records, not expanded occurrences
api.files.query(options?)	files:read	server-query FileNodes

All return:

{
  items,
  total,
  position,
  queryState,
  nextCursor, // opaque string or null
  accountId
}

limit: 0 asks only for the exact count and does not run /get. Otherwise the example uses 1–100 records per page (default 25), honors maxObjectsInGet, and caps the projected response at 256 KiB.

const { total } = await api.email.query({ limit: 0 });

// An application-owned checkpoint may survive a frame reload.
const page = await api.email.query({
  from: 'billing@example.test',
  limit: 50,
  cursor: savedCursor,
});
await savePageAndCheckpoint(page.items, page.nextCursor);

The host validates a narrow allowlist of filters. A cursor binds the calling plugin, server/login, primary resource account, method, normalized query and calendar time zone. It is a continuation, not an authentication token. Permissions and the active login are checked again after asynchronous reads. The public namespace is contacts; the bridge method continues to use the existing contact.* naming.

Completeness and compatibility

• The example requires the server's exact total and queryState. Unsupported or failed queries throw rather than looking like zero results.
• A changed query state, missing /get record or non-progressing page fails explicitly. A consumer restarts the aggregate instead of mixing pages from different result sets.
• A server-clamped short page is not treated as EOF; continuation advances by the actual returned ID count.
• Query state protects result membership/order, not immutable body bytes.
• Scope is one primary account for the resource, not a unified view or all delegated accounts. The returned accountId makes that boundary explicit.
• Calendar totals count series/standalone records. The existing occurrence-expanding API remains available for calendar views.
• Old Stalwart versions may omit directories from FileNode/query. The file total describes that query and is not advertised as an exhaustive directory-tree count.

The fork also includes api.files.readText(id, { maxBytes? }): it resolves a real FileNode, reads only that node's blob in its files account, refuses binary MIME types and returns a UTF-8 prefix with truncated. Its limit is 1–65,536 bytes (default 24,000). Whether file content deserves a separate permission from files:read is one point where upstream guidance would be useful.

Durable consumer ownership

The SDK remains a bounded read surface. The consumer owns cancellation, page/checkpoint persistence, reconnect, permission revocation, quotas and retry policy. My Flue consumer uses the original signed-in plugin frame as the host bridge and checkpoints page deltas with stable step.do names. It does not send whole collections to the model or introduce a second server-side login. That consumer integration is still in progress.

Verification and questions

The implementation was AI-assisted. The linked fork PR passed existing lint/typecheck, complete unit CI, Full container E2E, static Lite E2E, Stalwart Lite E2E, and performance checks. New tests cover 1,203 records over 401 server-clamped pages, count-only requests, changed/missing data, cursor scope, permissions and in-flight account changes. A Demo client implementation is included; no runtime dependency was added.

Feedback I would particularly appreciate:

1. Separate query methods versus extending existing methods with a page envelope?
2. Exact-count failure versus an explicit “total unavailable” result for servers that cannot calculate it?
3. Primary-account-only scope initially, versus an explicitly account-scoped contract for delegated accounts?
4. Naming and permission boundaries for stored-file metadata and bounded text reads?

I can adjust the fork example to a preferred upstream contract after discussion.

[yontani]

Consumer example update: the Flue integration is now merged in https://github.com/illuwa-soft/bulwark-agent/pull/9 (v0.4.0). Its production-build browser suite uses the shipped ZIP, opaque iframe, real Flue/SQLite DO and a controlled host/model fixture. It verifies a 105,743-record count-only result and a 51-record/17-page scan: after three committed pages, it removes the frame and SIGKILLs the owned Worker process group, then reconnects without resending the prompt or rereading committed pages. These counts are synthetic test data, not user mailbox counts. The tool checkpoints only small count/cursor/sample data, separates model-call limits from page execution, and reports explicit partial ranges/continuations. All 19 browser checks and 41 unit checks passed. This remains an implementation reference for discussion of the upstream API shape.