Repository navigation
Conversation
## Context Product catalog events need their own processing flow, but the record loop, the billable metric enrichment and the producers with their dead letter fallback are the same as for subscription events. ## Description Move these pieces out of the events processor into a pipeline package, unchanged, so a second flow can reuse them without depending on the subscription one. Kafka and database setup move to helpers for the same reason.
rsempe
force-pushed
the
catalog-events-processor-standalone
branch
from
October 9, 2026 12:58
ce62d8f to
44efed9
Compare
## Context Product catalog organizations send their events to their own topic, keyed by external_contract_id. Processing them inside the subscription processor would share its memory, CPU and failures, and mix contract lookups with the subscription flow. ## Description The processor now runs one pipeline per deployment, chosen by LAGO_EVENTS_PROCESSOR_PIPELINE. The default events pipeline is unchanged. The catalog pipeline consumes the catalog raw topic, resolves the contract serving each event, writes it to the catalog enriched topic, and sends it to price in advance when one of the contract's rate cards bills its metric in advance. Each pipeline caches only the tables it reads: the catalog one loads billable metrics, contracts, contract rate cards, rate cards and products, which Debezium now publishes. A dev service runs it under the catalog profile.
rsempe
force-pushed
the
catalog-events-processor-standalone
branch
from
October 9, 2026 13:15
44efed9 to
0ac56a3
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
An alternative to #819 and #826, which ran the catalog consumer inside the subscription processor. This version keeps the two flows apart: a catalog failure, backlog or memory spike can't hold back subscription events, and each flow can be scaled, deployed and eventually retired on its own.
Description
processors/pipeline. The events processor uses them as is.LAGO_EVENTS_PROCESSOR_PIPELINE=catalog_eventsrunsprocessors/catalog_processorinstead of the events processor, in a separate deployment from the same image.LAGO_KAFKA_CATALOG_RAW_EVENTS_TOPIC.LAGO_KAFKA_CATALOG_ENRICHED_EVENTS_TOPIC.eventspipeline is unchanged.catalog-events-processorruns under thecatalogprofile.Open points:
external_subscription_id. A dedicated topic would need a new API consumer.LAGO_USE_MEMORY_CACHE, the production connector must publishcontracts,contract_rate_cards,rate_cardsandproducts.extra/debezium_config.jsononly covers the dev stack. Without them, the catalog cache loads its snapshot and then never updates.