[bbaJay749]

What does this PR address?

TimeoutMiddleware schedules a call_later timer for every request but never cancels it. If the request finishes in time, the timer just sits in the event loop until traffic.timeout runs out (60s by default) and then fires on a future that is already done.

That would be harmless, except the pending TimerHandle keeps a few things alive the whole time: the waiter future, and the contextvars.Context that call_later copies when it schedules the callback. That context includes whatever the outer middlewares put there, e.g. the OpenTelemetry span. So every worker ends up holding on to every request from the last 60 seconds.

We ran into this while chasing long full-GC pauses on an internal service. At ~180 req/s per worker it adds up to roughly 240k extra live objects per worker (~22 per request). They survive the young-generation collections and land in gen 2, so every full collection has to walk all of them.

Related to #5642, which is a different problem in the same middleware. This PR doesn't touch that.

Repro

import asyncio
import gc

from bentoml._internal.server.http.traffic import TimeoutMiddleware


async def app(scope, receive, send):
    await send({"type": "http.response.start", "status": 200, "headers": []})
    await send({"type": "http.response.body", "body": b"ok"})


async def receive():
    return {"type": "http.request", "body": b"", "more_body": False}


async def send(message):
    pass


async def main(n: int = 1000) -> None:
    middleware = TimeoutMiddleware(app, timeout=60)
    scope = {"type": "http", "method": "GET", "path": "/", "headers": []}
    for _ in range(n):
        await middleware(scope, receive, send)
    gc.collect()
    timers = [o for o in gc.get_objects() if isinstance(o, asyncio.TimerHandle)]
    print(f"requests={n} live TimerHandle={len(timers)}")


asyncio.run(main())

On main this prints live TimerHandle=1000. With this change it prints 1: that's the last request's timer, already cancelled, and the loop drops it on its next iteration. (Python 3.12, default asyncio loop.)

Numbers from our service

Sync API, 6 workers, 1000 req/s in total. Two 10-minute runs per variant.

	Before	After
One full (gen-2) collection	173 ms	6.5 ms
One gen-0 collection	10 ms	4.8 ms
One gen-1 collection	30 ms	5 ms
P99.9 latency	60.6 ms	54.9 ms
P99.99 latency	193.5 ms	69.2 ms
Max latency (worse of the two runs)	246.5 ms	93.5 ms
CPU per request		-3.4%
RSS per worker	449 MB	400 MB

Fix

Keep the handle and cancel it in a finally. That covers normal completion, errors from the app, and cancellation. If the timeout has already fired, cancel() is a no-op, so the 504 path doesn't change.

Tests

Added tests/unit/_internal/server/test_traffic.py:

• test_timeout_middleware_cancels_timer: fails on main, passes with the fix.
• test_timeout_middleware_returns_504_on_timeout: checks the timeout response still works.

Before submitting:

• Does the Pull Request follow Conventional Commits specification naming?
• Does the code follow BentoML's code style, pre-commit run -a script has passed? (All hooks pass. buf-format and buf-lint were skipped, same as pre-commit.ci; this PR doesn't touch the gRPC protos.)
• Did you read through contribution guidelines and follow development guidelines?
• Did your changes require updates to the documentation? Have you updated those accordingly? (No docs change needed. User-facing behavior is unchanged.)
• Did you write tests to cover your changes?