Skip to content

[CELEBORN-2511] Fix labeled histograms sharing one reservoir and being missed in metrics count - #3908

Open
yew1eb wants to merge 1 commit into
apache:mainfrom
yew1eb:fix/metrics-histogram-label-keys
Open

yew1eb wants to merge 1 commit into
apache:mainfrom
yew1eb:fix/metrics-histogram-label-keys

Conversation

@yew1eb

@yew1eb yew1eb commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  1. AbstractSource.addHistogram computes the label-aware key metricNameWithLabel but
    registers the Dropwizard histogram under the bare name, so every labeled variant of the
    same histogram shares a single reservoir. It now registers under metricNameWithLabel,
    consistent with addTimer, addCounter and addGauge.
  2. AbstractSource.getAllMetricsNum now includes namedHistogram.size(), so the metrics
    capacity warning counts what getMetrics actually emits.

Why are the changes needed?

  • With the old code, adding a label (e.g. a disk label on PartitionFileSize) makes all
    labeled variants read the same aggregated reservoir, so they report identical
    Count/Max/percentiles. It is latent today because the only two histograms
    (FetchChunkTransferSize, PartitionFileSize) carry no labels yet. Registering under the
    labeled key also keeps a future removeHistogram(name, labels) consistent with the other
    metric types, which remove by the labeled key.
  • getMetrics emits histograms (fillInnerMetricsSnapshot(histograms(), ...)) but
    getAllMetricsNum ignores them, under-reporting the count used for the
    "number of metrics exceed the output metrics strings capacity" warning.

Does this PR resolve a correctness bug?

  • Yes

Does this PR introduce any user-facing change?

  • Yes

Labeled histograms report their own values instead of a shared aggregate, and the reported
total metric count grows by the number of histograms.

How was this patch tested?

Added test histogram with customized label to CelebornSourceSuite: it registers
PartitionFileSize with disk=disk1/disk=disk2, updates them with different values and
asserts each label reports its own _Max plus getAllMetricsNum == 2.

./build/mvn -pl common scalatest:test -DargLine= -Dsuites=org.apache.celeborn.common.metrics.source.CelebornSourceSuite

passes (7 tests); the new test fails when either fix is reverted.

@yew1eb
yew1eb force-pushed the fix/metrics-histogram-label-keys branch from 68496be to b026047 Compare October 28, 2026 12:52
@yew1eb yew1eb changed the title [CELEBORN-XXXX] Fix metric name/label mismatches for histograms, RPC metrics removal, and metrics count [CELEBORN-2511] Fix labeled histograms sharing one reservoir and being missed in metrics count Oct 28, 2026
@github-actions github-actions Bot added correctness Correctness bugfix module:common labels Oct 28, 2026
…g missed in metrics count

1. `AbstractSource.addHistogram` computes the label-aware key `metricNameWithLabel` but
   registers the Dropwizard histogram under the bare `name`, so every labeled variant of the
   same histogram shares a single reservoir. It now registers under `metricNameWithLabel`,
   consistent with `addTimer`, `addCounter` and `addGauge`.
2. `AbstractSource.getAllMetricsNum` now includes `namedHistogram.size()`, so the metrics
   capacity warning counts what `getMetrics` actually emits.

- With the old code, adding a label (e.g. a disk label on `PartitionFileSize`) makes all
  labeled variants read the same aggregated reservoir, so they report identical
  `Count`/`Max`/percentiles. It is latent today because the only two histograms
  (`FetchChunkTransferSize`, `PartitionFileSize`) carry no labels yet. Registering under the
  labeled key also keeps a future `removeHistogram(name, labels)` consistent with the other
  metric types, which remove by the labeled key.
- `getMetrics` emits histograms (`fillInnerMetricsSnapshot(histograms(), ...)`) but
  `getAllMetricsNum` ignores them, under-reporting the count used for the
  "number of metrics exceed the output metrics strings capacity" warning.

- [x] Yes

- [x] Yes

Labeled histograms report their own values instead of a shared aggregate, and the reported
total metric count grows by the number of histograms.

Added `test histogram with customized label` to `CelebornSourceSuite`: it registers
`PartitionFileSize` with `disk=disk1`/`disk=disk2`, updates them with different values and
asserts each label reports its own `_Max` plus `getAllMetricsNum == 2`.

    ./build/mvn -pl common scalatest:test -DargLine= -Dsuites=org.apache.celeborn.common.metrics.source.CelebornSourceSuite

passes (7 tests); the new test fails when either fix is reverted.
@yew1eb
yew1eb force-pushed the fix/metrics-histogram-label-keys branch from b026047 to 43f9a92 Compare October 28, 2026 13:19
@github-actions github-actions Bot removed the correctness Correctness bugfix label Oct 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant