Skip to content

[Feature] Add LLM token usage monitoring #49

Description

@mr-dibbs

Description of Problem

When I run LLM workflows, I want to see how many tokens each request uses, so I can control costs and catch problems early.

Right now there is no way to see token usage. The only option is the provider's billing page, which is delayed and only shows account totals. That means I can't:

  • Tell which feature, user, or workflow is spending the money
  • Catch a runaway loop before it runs up a big bill
  • Predict costs when scaling up or switching models

Potential Solutions

Proposed: built-in token tracking

  • Record input/output tokens for each request (use the provider's response data, or estimate if not available)
  • Group totals by model, session, and tags (user, feature, etc.)
  • Show the data in logs and a metrics endpoint
  • Add optional alerts when usage passes a set limit
    Benefits: real-time cost visibility, spend tied to specific workflows, early warning on runaway usage.

Drawbacks: small overhead per request; estimates may be off for some providers; more config to maintain.

Alternatives considered

  • External proxy/gateway: no code changes, but adds infrastructure and latency, and loses in-app context.
  • Manual logging: works, but everyone builds it differently and there's no shared standard.
  • Provider billing dashboards: free, but delayed and account-level only.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions