Skip to content

Fix orion start up race condition - #406

Merged
ArtemTetenkin merged 2 commits into
masterfrom
orion_kafka_start_race_condition_fix
Nov 6, 2026
Merged

ArtemTetenkin merged 2 commits into
masterfrom
orion_kafka_start_race_condition_fix

Conversation

@ArtemTetenkin

@ArtemTetenkin ArtemTetenkin commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Fix Kafka node registration race on Orion startup

Problem

When Orion starts, the cluster is registered with ClusterManager right after its sensors are started, but before the first KafkaBrokerSensor run (describeCluster) has returned. During that window, agent heartbeats hitting /register can create broker nodes via Cluster.updateNodeFromAgentHeartbeat, using the agent-reported NodeInfo as the node's currentNodeInfo.

Fix

  • KafkaCluster overrides updateNodeFromAgentHeartbeat and defers registration of new nodes until the first broker sensor run has populated the node map. Heartbeats for nodes that are already known (added by the sensor or restored from cluster state) are processed as before.
  • KafkaBrokerSensor calls markBrokerSensorCompleted() right after brokers are added to the node map, so a later failure (e.g. controller lookup) doesn't keep registrations blocked.
  • Deferred heartbeats return null (no command), which agents already handle; they retry on their next heartbeat. Metrics from the heartbeat are still published.
  • To avoid blocking registrations forever if Kafka is unreachable at startup, the gate is lifted after a 2-minute grace period.

@ArtemTetenkin
ArtemTetenkin requested a review from a team as a code owner November 4, 2026 22:29
@ArtemTetenkin
ArtemTetenkin merged commit 7ab5dc1 into master Nov 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants