feat: incremental planning and verified completion (#253)

* feat: require tool evidence for declared plan checks

* docs: explain acceptance checks and runtime evaluations

* feat: revise plans incrementally without dropping acceptance checks

* fix: resolve release CodeQL findings in planning and evals

Author: jkyberneees

Makefile

@@ -42,6 +42,10 @@ test-internal: ## Run only internal package tests (excludes cmd/odek)
 test-cmd: ## Run cmd/odek unit tests (env-gated E2E/sandbox suites skipped)
 	$(GO) test -short -count=1 -timeout 600s ./cmd/odek -skip 'TestE2E_|TestMCPE2E|TestSandbox'
 
+.PHONY: eval
+eval: ## Run deterministic local runtime evaluations (no credentials or external network)
+	$(GO) run ./cmd/odek-eval
+
 .PHONY: test-cli-serve
 test-cli-serve: ## Serve/WS/REST headless-run surface only (~1 min)
 	$(GO) test -short -count=1 -timeout 300s -run 'TestServe|TestWS|TestRestRun|TestPrompt|TestHandlePrompt' ./cmd/odek

README.md

@@ -201,7 +201,8 @@ odek run "@README.md what does this project do?"
 | [Extensions](docs/EXTENSIONS.md) | `odek-extension/v1` contract: MCP limits, artifact refs, event stream, external refs, budgets |
 | [Maintenance](docs/MAINTENANCE.md) | Storage janitor: retention, log rotation, `odek cleanup` |
 | [Extended Memory](docs/EXTENDED_MEMORY.md) | Atomic long-term memory layer (opt-in) |
-| [Planning](docs/PLANNING.md) | Plan tool, protected plan message, security model |
+| [Planning](docs/PLANNING.md) | Incremental plan revisions, acceptance checks, completion evidence |
+| [Runtime Evals](docs/EVALS.md) | Deterministic task scenarios, independent checks, JSON reports |
 | [Tool Selection](docs/TOOL_SELECTION.md) | Tool whitelist/blacklist guide and names reference |
 | [Daily Worker](docs/DAILY-WORKER.md) | Headless scheduled-worker patterns |
 | [Providers](docs/PROVIDERS.md) | go-llm-sdk registry, `--provider`, v2 knobs |
@@ -238,6 +239,7 @@ The full `Config` struct supports: `Provider`, `Providers`, `BaseURL` (selected-
 go test ./...                 # full suite, no setup required
 go test -race ./...           # also clean under the race detector
 go test -cover ./...          # per-package coverage report
+make eval                    # scripted runtime evaluations, no model credentials
 ODEK_E2E=1 go test ./cmd/odek/   # opt-in Docker / subprocess E2E suite
 ```
 

cmd/odek-eval/main.go

@@ -0,0 +1,24 @@
+// odek-eval runs deterministic, localhost-only runtime evaluations.
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"os"
+
+	"github.com/BackendStack21/odek/internal/eval"
+)
+
+func main() {
+	report := eval.Run(context.Background(), eval.Scenarios())
+	enc := json.NewEncoder(os.Stdout)
+	enc.SetIndent("", "  ")
+	if err := enc.Encode(report); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(2)
+	}
+	if report.Failed != 0 {
+		os.Exit(1)
+	}
+}

docs/CONFIG.md

@@ -408,7 +408,11 @@ Gives the agent a protected plan tool and a plan message that survives context t
 |-------|---------|---------|----------|-------------|
 | `planning.enabled` | `true` | `ODEK_PLANNING` | `--planning` / `--no-planning` | Enable the plan tool and protected plan message |
 | `planning.max_steps` | `12` | — | — | Plan steps allowed (clamped 1–50) |
-| `planning.max_render_chars` | `2000` | — | — | Cap on the rendered plan shown in the UI (clamped 200–8000) |
+| `planning.max_render_chars` | `2000` | — | — | Cap on the protected plan render (clamped 200–8000); checked or revised plans must fit in full, including reserved evidence space |
+
+Acceptance checks need no additional flag: declare them through `plan create`
+while planning is enabled. Large declarations may need a higher operator-set
+`max_render_chars`; project config cannot raise this cap.
 
 Feature behavior, verbs, and the security model are documented in [PLANNING.md](PLANNING.md).
 

docs/DEVELOPMENT.md

@@ -172,6 +172,15 @@ go test -v -count=1 ./cmd/odek/ -run "TestSubagent|TestDelegateTasks"
 
 Zero external test dependencies — tests use `httptest`, `testing`, and the standard library only.
 
+### Runtime evaluations
+
+Run `make eval` (or `go run ./cmd/odek-eval`) from the repository root. The
+eleven scripted scenarios use the production loop with localhost fixture tools
+and independent outcome checks. The command writes a JSON report and exits
+nonzero if a scenario assertion fails; expected task failures can still pass
+the scenario. No model credentials or external provider calls are needed.
+See [EVALS.md](EVALS.md) for report fields, limitations, and adding scenarios.
+
 ### Test layers
 
 | Layer | Runner | What's tested |

docs/EVALS.md

@@ -0,0 +1,56 @@
+# Runtime evaluation harness
+
+`make eval` runs `cmd/odek-eval`, a deterministic harness around the
+production `internal/loop.Engine`. It starts an OpenAI-compatible provider on
+localhost, feeds scripted assistant replies, and exposes small stateful
+fixture tools. No credentials, external network, or live model is used.
+
+Each scenario keeps its model messages separate from its oracle. The oracle
+checks fixture state and observed tool calls, so an assistant saying “done”
+cannot by itself make a task pass. The JSON report contains per-case scenario
+status, independently determined `task_success`, tool calls, synthetic
+scripted-provider token counts, and elapsed milliseconds. `false_completion_rate` is the fraction of
+cases whose oracle explicitly marked a success claim while required fixture
+state was absent; it is a regression signal, not a model-quality score.
+`cost_known` is always false for the shipped harness; it does not configure
+prices or estimate cost.
+
+The initial suite covers verified artifact work, a failed read followed by a
+false success claim, unrelated reads after a write, transient failure and
+recovery, cancellation, and plan acceptance checks for success, failed
+evidence, missing evidence, and incremental revision behavior. Negative cases can still be scenario passes
+when the oracle correctly records that the task did not succeed. The plan
+cases use the production `plan` tool and `PlanStore`, including the runtime
+incomplete marker for failed or missing evidence.
+
+The reusable `internal/eval.RunWithOptions` API accepts a per-case client
+factory. An application may use that adapter to compare a separately
+authorized real provider, but it must provide its own credentials, network
+policy, and model-message adapter. The shipped CLI intentionally does not
+evaluate live-model intelligence. Cost reporting stays unknown because the
+harness does not configure token prices.
+
+## Running and extending the suite
+
+```bash
+make eval
+# Save only the JSON report (without make's command echo):
+go run ./cmd/odek-eval > eval-report.json
+```
+
+Exit status is 0 when every scenario passes, 1 for scenario failures, and 2
+if the report cannot be encoded. The eleven-case baseline includes a deliberate
+unguarded false-success control: all scenarios pass while
+`false_completion_rate` is 1/11 (about 0.091). That expected control is not a failure of
+the checked-plan guard or a live-model benchmark.
+
+Add a case to `internal/eval.Scenarios` with fresh fixture state, scripted
+responses, tools, and an independent oracle. Assert the required state and
+observed tool outcomes, including expected failures; do not accept a success
+claim as proof. `task_success` answers whether the fixture task was completed;
+`scenario_passed` answers whether the runtime behaved as the test expected.
+Add regression assertions in `internal/eval/eval_test.go`, then run:
+
+```bash
+go test -count=1 -timeout=120s ./internal/eval ./internal/loop
+```