[pistachiobaby]

When a oneshot pod dies mid-request, the router's 502 now has a JSON body saying why. Standard functions keep the empty 502. On local OrbStack, 4 real OOM kills through the router came back OOMKilled with the 256Mi limit within 1.1 s, and a mid-request eviction came back EvictionByEvictionAPI. The full suite (1,267 tests) passes locally and on CI's kind cluster.

The problem

In the 30 days to 15 Oct 2026, 38 operation-runner runCommand requests ended in an empty 502 (failed to forward request, EOF). 19 matched kernel OOM kills; the other 19 have no known cause. The pod's container status records why (OOMKilled, exit 137), but the router releases the pod right after the error, and deleting it discards that status.

Now the router calls ReleaseInstance with report_termination. The controller reads the pod every 250 ms for up to --instance-termination-wait (5 s), returns the reason, then deletes it. It waits because the kubelet notices a dead container on its next relist (every 1 s). The controller reads because the router has no pod RBAC (Gadget's production role grants pods *). The body:

{"error":"instance operation-runner-…-x2k8p stopped before it responded: container \"runner\" ran out of memory and was killed at its 6Gi limit (exit code 137)","reason":"OOMKilled","instance":"operation-runner-…-x2k8p","container":"runner","exitCode":137,"memoryLimitBytes":6442450944}

• reason: OOMKilled, PodTerminated, ContainerExited, or Unknown. A liveness-probe kill shows as ContainerExited, since Kubernetes records probes only as Events.
• container, exitCode: on OOMKilled and ContainerExited. memoryLimitBytes: on OOMKilled.
• terminatedBy: on PodTerminated when known. HeartbeatTimeout if skipper deleted the pod, else Kubernetes' DisruptionTarget reason (EvictionByEvictionAPI, TerminationByKubelet).

Why annotate heartbeat-timeout deletes

A heartbeat timeout deletes every pod of a oneshot function, running commands included. ReleaseInstance can land on any controller, so the deleting one first writes skipper/terminated-by: HeartbeatTimeout on the pod: one extra patch per orphaned pod.

Not verified

• The kubelet's delay on a GKE node under memory pressure. Locally it was 0.4 to 1.1 s over 10 kills; past 5 s the body says Unknown.
• Heartbeat-timeout deletes and kubelet node-pressure evictions are unit-tested only.

Next: the Gadget api reads body.reason in OperationRunnerClient, a follow-up to gadget-inc/gadget#20960.

🤖 Generated with Claude Code