Skip to content

feat(events-processor): run catalog events as a separate pipeline - #827

Draft
rsempe wants to merge 2 commits into
mainfrom
catalog-events-processor-standalone
Draft

rsempe wants to merge 2 commits into
mainfrom
catalog-events-processor-standalone

Conversation

@rsempe

@rsempe rsempe commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Context

An alternative to #819 and #826, which ran the catalog consumer inside the subscription processor. This version keeps the two flows apart: a catalog failure, backlog or memory spike can't hold back subscription events, and each flow can be scaled, deployed and eventually retired on its own.

Description

  • Shared building blocks (first commit, no behaviour change): the record loop with its retry and dead-letter rules, the billable metric enrichment, and the producers move to processors/pipeline. The events processor uses them as is.
  • Catalog pipeline: LAGO_EVENTS_PROCESSOR_PIPELINE=catalog_events runs processors/catalog_processor instead of the events processor, in a separate deployment from the same image.
    • It consumes only LAGO_KAFKA_CATALOG_RAW_EVENTS_TOPIC.
    • It resolves the contract serving each event and writes the event to LAGO_KAFKA_CATALOG_ENRICHED_EVENTS_TOPIC.
    • It sends the event to be priced in advance when a contract rate card bills its metric in advance.
    • The default events pipeline is unchanged.
  • Cache per pipeline: the catalog pipeline loads billable metrics, contracts, contract rate cards, rate cards and products; the events pipeline keeps its current tables. Debezium now publishes the catalog tables.
  • Dev stack: catalog-events-processor runs under the catalog profile.

Open points:

  • Advance pricing: catalog events are still sent to the subscription charged-in-advance topic, with the contract's external id in external_subscription_id. A dedicated topic would need a new API consumer.
  • Deployment: the Helm chart and lago-deploy need a second processor deployment with the catalog variables.
  • Debezium: with LAGO_USE_MEMORY_CACHE, the production connector must publish contracts, contract_rate_cards, rate_cards and products. extra/debezium_config.json only covers the dev stack. Without them, the catalog cache loads its snapshot and then never updates.

## Context

Product catalog events need their own processing flow, but the record
loop, the billable metric enrichment and the producers with their dead
letter fallback are the same as for subscription events.

## Description

Move these pieces out of the events processor into a pipeline package,
unchanged, so a second flow can reuse them without depending on the
subscription one. Kafka and database setup move to helpers for the same
reason.
@rsempe
rsempe force-pushed the catalog-events-processor-standalone branch from ce62d8f to 44efed9 Compare October 9, 2026 12:58
## Context

Product catalog organizations send their events to their own topic,
keyed by external_contract_id. Processing them inside the subscription
processor would share its memory, CPU and failures, and mix contract
lookups with the subscription flow.

## Description

The processor now runs one pipeline per deployment, chosen by
LAGO_EVENTS_PROCESSOR_PIPELINE. The default events pipeline is
unchanged. The catalog pipeline consumes the catalog raw topic,
resolves the contract serving each event, writes it to the catalog
enriched topic, and sends it to price in advance when one of the
contract's rate cards bills its metric in advance.

Each pipeline caches only the tables it reads: the catalog one loads
billable metrics, contracts, contract rate cards, rate cards and
products, which Debezium now publishes. A dev service runs it under the
catalog profile.
@rsempe
rsempe force-pushed the catalog-events-processor-standalone branch from 44efed9 to 0ac56a3 Compare October 9, 2026 13:15

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant