Repository navigation
Conversation
Modify ChangePartitionManager and LifecycleManager
5590ef0 to
0dffcf6
Compare
|
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
CIP document updated |
|
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This issue was closed because it has been staled for 10 days with no activity. |
turboFei
changed the title
|
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This issue was closed because it has been staled for 10 days with no activity. |
|
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This issue was closed because it has been staled for 10 days with no activity. |
|
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This issue was closed because it has been staled for 10 days with no activity. |
…nd refocus design doc on CIP-20 Rename the parallel-write feature to adaptivePartitionWriteParallelism: - celeborn.client.shuffle.adaptivePartitionWriteParallelism.enabled / .maxLocations / .hotWindow (config keys, constants, getters, locals) - ChangePartitionManagerParallelWriteSuite renamed to ChangePartitionManagerAdaptiveParallelismSuite - Regenerate docs/configuration/client.md Rewrite the design doc: drop the Uniffle introduction, add a deep analysis of CIP-20 / PR apache#3260 based on its actual code (event-count x assumed-bytes rate signal vs our measured fillTime; the static expectedWorkerSpeed magic number; epoch-semantics reuse and its invasive fallout; closed by stale bot without technical rejection), and rebuild the comparison as a two-way table against CIP-20 only. Remove dynamic-write-parallelism-comparison.md accordingly. Compile, spotless and the config-docs golden check pass.
…nd refocus design doc on CIP-20 Rename the parallel-write feature to adaptivePartitionWriteParallelism: - celeborn.client.shuffle.adaptivePartitionWriteParallelism.enabled / .maxLocations / .hotWindow (config keys, constants, getters, locals) - ChangePartitionManagerParallelWriteSuite renamed to ChangePartitionManagerAdaptiveParallelismSuite - Regenerate docs/configuration/client.md Rewrite the design doc: drop the Uniffle introduction, add a deep analysis of CIP-20 / PR apache#3260 based on its actual code (event-count x assumed-bytes rate signal vs our measured fillTime; the static expectedWorkerSpeed magic number; epoch-semantics reuse and its invasive fallout; closed by stale bot without technical rejection), and rebuild the comparison as a two-way table against CIP-20 only. Remove dynamic-write-parallelism-comparison.md accordingly. Compile, spotless and the config-docs golden check pass.
|
Hi @ErikFang @mridulm @turboFei @RexXiong — I've been working on the same problem and have submitted CIP-24 (#3843), which shares the goal of adaptive partition write parallelism but takes a different approach to the control loop (LifecycleManager measures actual fill time per location rather than estimating from event counts, and the active set is propagated through the existing revive RPC). Your earlier work on CIP-20 was a valuable reference for this design. The proposal is under discussion on dev@: https://lists.apache.org/thread/fvy5jor59mxqp15v8xq2d5xw64yb71gs — would love to hear your thoughts there or on the PR. |
6 participants
What changes were proposed in this pull request?
enhance celeborn to
CIP-20: Dynamically optimize shuffle write parallelism
https://docs.google.com/document/d/1CqFswIOP5nR8Cy2THo8tELOwVxUf0ZP_8pDjaYj2HGc/edit?usp=sharing
Why are the changes needed?
TBD
Does this PR introduce any user-facing change?
TBD
How was this patch tested?
TBD