Repository navigation
Conversation
yew1eb
marked this pull request as draft
October 13, 2026 11:28
…ffle metrics Extend the CELEBORN-2415 Spark UI skeleton into a full observability tab: - Instrument the client write/read paths (PushState, InFlightRequestTracker, ReadStreamStats) and report the aggregates to the driver through the MapperEnd RPC and a new ReportShuffleReadMetrics RPC. - Post custom CelebornEvents (build info, shuffle assignment, fallback, reassign, write/read metrics) from the driver into the listener bus via the status queue, so they persist into the event log and replay on the HistoryServer. - Fuse everything into a single CelebornListener that also consumes native TaskMetrics, aggregating per-shuffle write/read metrics and per-worker push/read costs into the KVStore. - Render a Uniffle-style page: headline stat tiles, per-shuffle details table (joined assignments + write + read metrics with derived speed/share columns), vertical write/read time breakdowns, per-worker tables with avg RTT, build info, fallback stats and formatted timestamps. Co-Authored-By: Claude <noreply@anthropic.com>
The Spark UI extension read its configs by bare keys from SparkConf: celeborn.client.spark.ui.enabled in CelebornUIUtils.isUIEnabled and celeborn.client.spark.ui.retainedShuffles in CelebornListener. SparkConf matches keys exactly, so user-set spark.celeborn.* values never applied: the listener and tab were still registered when the extension was disabled (breaking the documented "zero-overhead when off"), and retainedShuffles was stuck at the default. Both now read through CelebornUIUtils.celebornConf, a local spark.celeborn.* -> CelebornConf conversion (SparkUtils.fromSparkConf does the same but its class init drags in spark-sql's UnsafeRowSerializer, which is not on this module's classpath and absent in RDD-only applications), so the entry defaults and the retainedShuffle alias apply. CelebornPlugin's buildInfoMap is switched to the same helper to avoid the spark-sql class init there as well. Co-Authored-By: Claude <noreply@anthropic.com>
yew1eb
force-pushed
the
spark-ui-enhancement
branch
from
October 13, 2026 11:37
6455ebd to
a57bba7
Compare
github-actions
Bot
added
module:client
module:spark
kind:documentation
module:common
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Gate the Celeborn Spark UI extension on the new
celeborn.client.spark.ui.enabledconfig so the documented "zero-overhead when off" actually holds.Today
CelebornPluginregistersCelebornListenerand attaches the Celeborn tab unconditionally: settingspark.celeborn.client.spark.ui.enabled=falsenever disables anything. And a bare-keySparkConf.getBoolean/getIntlookup would not help either, because SparkConf matches keys exactly while user-set values carry thespark.celeborn.prefix — so the flag must be resolved throughCelebornConf.This PR:
celeborn.client.spark.ui.enabledentry (categories: client, defaulttrue) with its accessor and the regenerateddocs/configuration/client.mdrow;CelebornUIUtils.celebornConf, a localspark.celeborn.* -> CelebornConfconversion, and reads the flag through theCelebornConfentry (never a bare-key lookup). The conversion is local to thespark-3-uimodule becauseSparkUtils.fromSparkConf's class init drags in spark-sql'sUnsafeRowSerializer, which is not on this module's classpath (spark-core only) and is absent in RDD-only applications;CelebornDriverPlugin.init(listener registration) andregisterMetrics(tab attachment) on the flag, together with Spark's ownspark.ui.enabled.Why are the changes needed?
The extension could not be turned off, so the listener stayed on the status queue and metrics kept being collected even when the user disabled the UI extension. Fixes CELEBORN-2477.
Does this PR introduce any user-facing change?
Yes:
celeborn.client.spark.ui.enabled=falsenow disables the Celeborn Spark UI extension (listener + tab). Default behavior is unchanged (enabled).How was this patch tested?
uiGatingHonorsSparkCelebornPrefixcase inCelebornListenerSuite: default true,spark.celeborn.client.spark.ui.enabled=falsedisables,spark.ui.enabled=falsedisables.mvn -Pspark-3.5 -pl client-spark/spark-3-ui -am clean testpasses (incl. theConfigurationSuitegolden-file check for the new doc row).