Skip to content

fix(agent): a command's timeout never outlasts the turn's time limit - #8424

Open
kovtcharov wants to merge 3 commits into
amd:mainfrom
kovtcharov:kalin/cap-tool-timeout-at-time-limit
Open

kovtcharov wants to merge 3 commits into
amd:mainfrom
kovtcharov:kalin/cap-tool-timeout-at-time-limit

Conversation

@kovtcharov

@kovtcharov kovtcharov commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

A turn run with --time-limit could spend most of its time inside one command and end with nothing written. A script left at its 600 s default timeout ran straight past a 720 s limit's stop point; the turn was then killed before it wrote its result. Now a shell or Python command the agent starts gets a timeout no longer than the time left before the stop point, and a command with under 5 s left is refused with a message to write the result instead. Turns with no time limit are unchanged, and no tool description changes.

Checked end to end: under --time-limit 120, a command that sleeps 600 s was cut off at 110 s and the agent still wrote its result file before the limit.

Test plan

  • python -m pytest tests/unit/agents/test_turn_time_limit.py tests/unit/test_run_python_timeouts.py
  • gaia -p "run: python -c 'import time; time.sleep(600)', then write result.txt saying what happened" --time-limit 120 --full-access cuts the command off before the limit and still writes result.txt

A turn with a time limit could spend most of it inside one command: a script
left at its 600 s class default ran past a 720 s limit's stop point and was
killed with nothing written. A shell or Python run the agent starts now gets a
timeout no longer than the time left before the stop point, and a run with
under 5 s left is refused with a message to write the result instead.

Turns with no time limit are unchanged.
@github-actions github-actions Bot added documentation Documentation changes cli CLI changes tests Test changes agents labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Verdict: Approve with suggestions

With --time-limit set, a shell or Python command now gets a timeout that ends before the turn's stop point, and a command started with under 5 s left is refused with a "write your result now" message. This fixes the reported failure, where a command with a 600 s default ran past a 720 s limit and the turn was killed before writing anything. The change is small and well tested, and turns with no time limit are unchanged.

Two gaps are left on nearby paths, and neither blocks the merge:

  • The shell tool sets the cut-down timeout before it waits out its rate limit. That wait can last up to 60 s, so a command can still run into the stop point. Moving the cap after the wait fixes it.
  • Follow-up: the agent's sleep tool doesn't respect the time limit yet. A 300 s wait late in a turn still eats the time the agent needs to write its result.

Real-world evidence

N/A: no evidence bundle was produced for this run. This verdict rests on static review and the PR's unit tests. The gaia -p … --time-limit 120 item in the test plan hasn't been shown on the PR. Pasting its output would close that gap.

🔍 Technical details

Issues

🟢 Cap is computed before the rate-limit wait (src/gaia/agents/tools/shell_tools.py:8182-8186)
_pace_rate_limit() can sleep up to max_rate_limit_wait_seconds (60 s) after the cap is taken, whenever bypass is false. That covers gaia -p --dangerously-skip-permissions without --full-access. The capped timeout is then stale by however long the wait took, and the command can run past the stop point. The hard stop still ends it, but the turn loses the write-up time this PR is meant to protect. Move the three cap lines below the rate-limit block so the cap reads the clock just before the command launches.

🟡 Follow-up: sleep tool ignores the turn time limit (src/gaia/agents/tools/wait_tools.py:138-154)
This is the same failure on a sibling path, and was found by grepping for other long-running tools. _wait_interrupted checks only cancel signals, never _time_limit_passed() or time_capped_run_seconds. Suggested issue: "sleep tool should end before a --time-limit turn's stop point".

🟢 Docstring disagrees with its own comment (tests/unit/test_run_python_timeouts.py:419)
The docstring says "3 s before its stop point", but the setup gives 8 s (90 - 82).

    """A 120 s default must not outlast a turn with 8 s before its stop point."""

Sibling paths checked: both Python entry points in chat_agent.py (run_python, execute_python_file) and run_shell_command are capped. time_limit_s is set only by print_mode.py, so the Agent UI and TUI are unaffected. The getattr guard covers shell hosts that aren't an Agent. The doc row in cli.mdx is the only place that describes this behaviour, and it was updated.

Strengths

  • One helper on Agent and one shared OUT_OF_TIME refusal (built on NOT_EXECUTED) serve every caller, so the verifier sees "nothing ran" in the same form whichever tool refused.
  • The tests cover the arithmetic table, the shell path end to end through the registered tool, and a real subprocess that gets cut short. That last one shows the cap actually shortens the run, not just that the helper was called.
  • The new refusal tells the model exactly what to do next ("write the result you have now"), rather than giving it a bare error.

Review follow-ups to the time-limit cap:

- The cap assumed the stop was at 90% of the limit; slow steps move it
  earlier. The loop and the cap now read one stop point.
- The cap is applied after the shell's rate-limit wait, which is time the
  turn has also spent.
- sleep and wait_for_condition are capped the same way.
- A run the limit cut short says so, in place of advice to rescope it or
  ask for a larger timeout.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents cli CLI changes documentation Documentation changes tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant