Skip to content

fix: skip empty segment ids for cdp_member_segment_aggs sinks - #4454

Merged
epipav merged 2 commits into
mainfrom
fix/member-aggregates-sink-empty-segment-id
Aug 10, 2026
Merged

fix: skip empty segment ids for cdp_member_segment_aggs sinks#4454
epipav merged 2 commits into
mainfrom
fix/member-aggregates-sink-empty-segment-id

Conversation

@epipav

@epipav epipav commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

Signed-off-by: anilb <epipav@gmail.com>
Copilot AI balanced review requested due to automatic review settings August 8, 2026 11:02
@cursor

cursor Bot commented Aug 8, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes what aggregate rows reach Kafka for member/org segment sinks; correct for bad data but could hide edge cases if empty ids were relied on downstream.

Overview
Filters out empty parentId / grandparentId on CDP member segment aggregate pipes (daily “changed” sinks and bucket backfillers) so parent/grandparent rollups and segment selection no longer emit rows keyed by blank segment ids to memberSegmentsAgg_sink.

The same non-empty hierarchy id checks are added on organization bucket backfill parent/grandparent nodes (s.parentId <> '', s.grandparentId <> ''), and leaf org backfill now always applies organizationId <> '' instead of only when bucket_id is set. Optional bucket_id filtering is unchanged but moved to AND clauses on a always-present WHERE, so base exclusions still run when bucketing is omitted.

Reviewed by Cursor Bugbot for commit 6ee609a. Bugbot is set up for automated code reviews on this repo. Configure here.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prevents empty hierarchy segment IDs from reaching member and organization aggregate sinks.

Changes:

  • Filters empty parent and grandparent segment IDs.
  • Applies filters to backfill and daily changed-aggregate sinks.
  • PR title still needs the CI-required (CM-XXX) JIRA key.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
cdp_organization_aggregates_bucket_backfiller_sink.pipe Filters invalid organization and hierarchy IDs.
cdp_member_aggregates_parent_segments_bucket_backfiller_sink.pipe Excludes empty parent IDs.
cdp_member_aggregates_grandparent_segments_bucket_backfiller_sink.pipe Excludes empty grandparent IDs.
cdp_member_aggregates_changed_parent_segments_sink.pipe Filters empty parent IDs from daily exports.
cdp_member_aggregates_changed_grandparent_segments_sink.pipe Filters empty grandparent IDs from daily exports.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@epipav epipav self-assigned this Aug 8, 2026
@epipav
epipav requested a review from gaspergrom August 8, 2026 17:15
Copilot AI review requested due to automatic review settings August 10, 2026 14:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

@epipav
epipav merged commit 47d0289 into main Aug 10, 2026
16 checks passed
@epipav
epipav deleted the fix/member-aggregates-sink-empty-segment-id branch August 10, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants