Description of Problem
When I run LLM workflows, I want to see how many tokens each request uses, so I can control costs and catch problems early.
Right now there is no way to see token usage. The only option is the provider's billing page, which is delayed and only shows account totals. That means I can't:
- Tell which feature, user, or workflow is spending the money
- Catch a runaway loop before it runs up a big bill
- Predict costs when scaling up or switching models
Potential Solutions
Proposed: built-in token tracking
- Record input/output tokens for each request (use the provider's response data, or estimate if not available)
- Group totals by model, session, and tags (user, feature, etc.)
- Show the data in logs and a metrics endpoint
- Add optional alerts when usage passes a set limit
Benefits: real-time cost visibility, spend tied to specific workflows, early warning on runaway usage.
Drawbacks: small overhead per request; estimates may be off for some providers; more config to maintain.
Alternatives considered
- External proxy/gateway: no code changes, but adds infrastructure and latency, and loses in-app context.
- Manual logging: works, but everyone builds it differently and there's no shared standard.
- Provider billing dashboards: free, but delayed and account-level only.
Description of Problem
When I run LLM workflows, I want to see how many tokens each request uses, so I can control costs and catch problems early.
Right now there is no way to see token usage. The only option is the provider's billing page, which is delayed and only shows account totals. That means I can't:
Potential Solutions
Proposed: built-in token tracking
Benefits: real-time cost visibility, spend tied to specific workflows, early warning on runaway usage.
Drawbacks: small overhead per request; estimates may be off for some providers; more config to maintain.
Alternatives considered