Skip to content

feat(account-tree): add support for attribution labels in GO processor - #824

Open
lovrocolic wants to merge 7 commits into
mainfrom
event-processor-attribution-labels
Open

lovrocolic wants to merge 7 commits into
mainfrom
event-processor-attribution-labels

Conversation

@lovrocolic

@lovrocolic lovrocolic commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Context

Account tree attributes usage to custom types (users, teams, API keys...). The read path groups events_enriched by attribution_labels, which nothing fills yet.

Description

The events-processor now loads the organization's usage_attribution_types and stamps each enriched event with attribution_labels: {type_code => value}, taken from the first of the type's attribution_keys found in the properties. Events are expected to carry the full ancestor chain. Missing, nested or too-long (>255 chars) values are skipped, and the field is omitted when empty, so organizations without types are unaffected.

Comment thread extra/debezium_config.json
@vincent-pochet

Copy link
Copy Markdown
Contributor

Bug: numeric attribution values get scientific-notation labels (events-processor/models/usage_attribution_types.go:101)

  • Properties are decoded with json.Unmarshal into map[string]any, so every number becomes a float64. fmt.Sprintf("%v", ...) then prints 1234567 as 1.234567e+06.
  • Result: attribution_labels['user'] IN ('1234567') never matches. Any integer ≥ 1,000,000 drops out of the account-tree filters.
  • Fix: for float64, use strconv.FormatFloat(v, 'f', -1, 64), or decode with json.Number.

Nit: the struct tags in models/event.go mix tabs and spaces. Run gofmt.

@lovrocolic

Copy link
Copy Markdown
Contributor Author

Bug: numeric attribution values get scientific-notation labels (events-processor/models/usage_attribution_types.go:101)

  • Properties are decoded with json.Unmarshal into map[string]any, so every number becomes a float64. fmt.Sprintf("%v", ...) then prints 1234567 as 1.234567e+06.
  • Result: attribution_labels['user'] IN ('1234567') never matches. Any integer ≥ 1,000,000 drops out of the account-tree filters.
  • Fix: for float64, use strconv.FormatFloat(v, 'f', -1, 64), or decode with json.Number.

Nit: the struct tags in models/event.go mix tabs and spaces. Run gofmt.

Thanks! Applied fix!

Comment thread events-processor/cache/cache.go Outdated
return nil
})

errGroup.Go(func() error {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cache is deprecated and will be removed, you can remove this part

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed 👍

@@ -0,0 +1,85 @@
package cache

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

@lovrocolic lovrocolic Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed file

@@ -0,0 +1,106 @@
package cache

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed file

@lovrocolic
lovrocolic force-pushed the event-processor-attribution-labels branch from 247078a to 07d5eea Compare October 16, 2026 08:28
@vincent-pochet

vincent-pochet commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Here are few finding from a Claude code review.
The caching logic for 1 can help with performance, but it might be a bit more work.

# Severity Where Issue Suggestion
1 🔴 High enrichment_service.go Uncached usage_attribution_types query on every event, even for orgs without the feature Cache per org with a TTL
2 🔴 High enrichment_service.go A failed attribution lookup fails the enrichment: billing usage and pay-in-advance get blocked (retries, then DLQ) Fall back to no labels and report the error
3 🟠 Medium usage_attribution_types.go Numeric IDs decoded as float64 lose precision above 2^53, so nearby IDs merge Decode with UseNumber
4 🟠 Medium enrichment_service.go Labels are built after evaluateExpression overwrites Properties[bm.FieldName], so a label can carry the computed value Build labels from the raw properties
5 🟠 Medium usage_attribution_types.go Labels keyed by code: a deleted type's code can be reused and old/new labels merge in ClickHouse Key by type ID, or freeze codes of deleted types
6 🟡 Low usage_attribution_types.go A child label is stamped when its parent label is missing, so tree totals don't add up Drop the child label when an ancestor is missing
7 🟡 Low usage_attribution_types.go Whitespace-only values become labels that the API's presence validation rejects Check with strings.TrimSpace before the empty check
8 ⚪ Nit usage_attribution_types.go SELECT * on the hot path, no ORDER BY Select only the columns used

@lovrocolic

Copy link
Copy Markdown
Contributor Author

Here are few finding from a Claude code review. The caching logic for 1 can help with performance, but it might be a bit more work.

Severity Where Issue Suggestion

1 🔴 High enrichment_service.go Uncached usage_attribution_types query on every event, even for orgs without the feature Cache per org with a TTL
2 🔴 High enrichment_service.go A failed attribution lookup fails the enrichment: billing usage and pay-in-advance get blocked (retries, then DLQ) Fall back to no labels and report the error
3 🟠 Medium usage_attribution_types.go Numeric IDs decoded as float64 lose precision above 2^53, so nearby IDs merge Decode with UseNumber
4 🟠 Medium enrichment_service.go Labels are built after evaluateExpression overwrites Properties[bm.FieldName], so a label can carry the computed value Build labels from the raw properties
5 🟠 Medium usage_attribution_types.go Labels keyed by code: a deleted type's code can be reused and old/new labels merge in ClickHouse Key by type ID, or freeze codes of deleted types
6 🟡 Low usage_attribution_types.go A child label is stamped when its parent label is missing, so tree totals don't add up Drop the child label when an ancestor is missing
7 🟡 Low usage_attribution_types.go Whitespace-only values become labels that the API's presence validation rejects Check with strings.TrimSpace before the empty check
8 ⚪ Nit usage_attribution_types.go SELECT * on the hot path, no ORDER BY Select only the columns used

Thanks @vincent-pochet! Regarding cache, Jeremy told me in earlier comments that it is deprecated. First version of this PR included cache for attribution types - #824 (comment)

@lovrocolic

lovrocolic commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

@vincent-pochet Applied fixes. Per our discussion, I removed caching but also extended billable metrics select (that runs anyway) with additional bool attribute. If organization does not have attribution types which is checked from the bool value, we will skip the attribution types fetch and do an early exit.
Having this bool on billable metrics could feel unclear, but we are avoiding extra query that way. Hope it works.

Regarding #5, we always fetch active types and this is also checked in the read path. Reusing the same code could have an impact, but if needed we can handle it in the api repo. #6 is also handled, since we display unattributed usage for all ingested usage with invalid attribution parameters.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants