You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dfde5de
Browse filesBrowse the repository at this point in the historyBrowse files
[CELEBORN-2451] Make DRA safe under AUTO fallback by tying supportsReliableStorage to fallback policy
### What changes were proposed in this pull request?
- Tie `supportsReliableStorage()` to the fallback policy: `true` only for
`NEVER`, `false` for `AUTO`/`ALWAYS` (was keyed on deprecated
`forceFallback`, so it returned `true` under `AUTO`).
- Make `SparkShuffleManager` DRA warnings policy-aware; log the
`registerShuffle` fallback at `error` only when shuffle tracking is also off.
### Why are the changes needed?
- Fallback is per-shuffle, but `supportsReliableStorage()` is one global flag.
Under `AUTO` a shuffle can fall back to local-disk `SortShuffleManager`;
that output dies with a DRA-reclaimed executor, causing `FetchFailedException`.
Nothing prevents the unsafe config today.
- Reporting `false` under `AUTO`/`ALWAYS` lets Spark's `ExecutorAllocationManager`
fail fast unless a shuffle service, shuffle tracking, or decommissioning is on,
so `AUTO` is safe with `shuffleTracking.enabled=true`; `NEVER` keeps the fast path.
- Trade-off: under `AUTO`, `reliableStorage=false` makes Spark recompute a lost
executor's map stages even though Celeborn still holds the data; `NEVER` avoids
this. Removing it needs a Spark-side change (per-shuffle reliable-storage
granularity), tracked as a follow-up.
### Does this PR resolve a correctness bug?
- [x] Yes
### Does this PR introduce _any_ user-facing change?
- [x] Yes: with `AUTO` (default), DRA without shuffle service and without shuffle
tracking now fails fast at startup. Enable tracking or set `NEVER`.
### How was this patch tested?
- New `testReliableStorageFollowsFallbackPolicy`; `-Dtest=SparkShuffleManagerSuite` passes (5/5), spotless clean.
Closes#3834 from venkata91/dra-auto-fallback-reliable-storage.
Authored-by: Venkata Krishnan Sowrirajan <venkat.sowrirajan@gmail.com>
Signed-off-by: Nicholas Jiang <programgeek@163.com>
# Support ShuffleManager when defined in user jars
327
328
# Required Spark version < 4.0.0 or without SPARK-45762, highly recommended to false for ShuffleManager in user-defined jar specified by --jars or spark.jars
328
329
spark.executor.userClassPathFirst false
329
330
```
330
331
332
+
With DRA enabled, `AUTO` and `ALWAYS` fallback policies require shuffle tracking unless an external
333
+
shuffle service or Spark shuffle decommissioning is configured. Shuffle decommissioning requires
334
+
`spark.decommission.enabled=true` and `spark.storage.decommission.shuffleBlocks.enabled=true`;
335
+
also enable `spark.storage.decommission.enabled` to migrate stored blocks.
336
+
To keep all shuffle data on Celeborn, set `spark.celeborn.client.spark.shuffle.fallback.policy=NEVER`.
337
+
With `NEVER`, the application fails rather than falling back when Celeborn cannot serve a shuffle.
338
+
Shuffle tracking can then be disabled to release idle executors promptly, but only when Spark's
339
+
dynamic allocation check still passes without it: either use Spark 3.5 or later with
340
+
`spark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO`, or
341
+
enable Spark shuffle decommissioning as described above. Otherwise, for example on Spark 3.4 without
342
+
an external shuffle service or shuffle decommissioning, Spark rejects dynamic allocation at startup.
343
+
331
344
Shuffle OSS storage related configurations:
332
345
```properties
333
346
# If you are using Celeborn for shuffle OSS storage, these settings will be needed.
# Support ShuffleManager when defined in user jars
215
216
# Required Spark version < 4.0.0 or without SPARK-45762, highly recommended to false for ShuffleManager in user-defined jar specified by --jars or spark.jars
216
217
spark.executor.userClassPathFirst false
217
218
```
218
219
220
+
With DRA enabled, `AUTO` and `ALWAYS` fallback policies require shuffle tracking unless an external
221
+
shuffle service or Spark shuffle decommissioning is configured. Shuffle decommissioning requires
222
+
`spark.decommission.enabled=true` and `spark.storage.decommission.shuffleBlocks.enabled=true`;
223
+
also enable `spark.storage.decommission.enabled` to migrate stored blocks.
224
+
To keep all shuffle data on Celeborn, set `spark.celeborn.client.spark.shuffle.fallback.policy=NEVER`.
225
+
With `NEVER`, the application fails rather than falling back when Celeborn cannot serve a shuffle.
226
+
Shuffle tracking can then be disabled to release idle executors promptly, but only when Spark's
227
+
dynamic allocation check still passes without it: either use Spark 3.5 or later with
228
+
`spark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO`, or
229
+
enable Spark shuffle decommissioning as described above. Otherwise, for example on Spark 3.4 without
230
+
an external shuffle service or shuffle decommissioning, Spark rejects dynamic allocation at startup.
231
+
219
232
## Deploy Flink client
220
233
221
234
**Important: Only Flink batch jobs are supported for now. Due to the Shuffle Service in Flink is cluster-granularity, if you want to use Celeborn in a session cluster, it will not be able to submit both streaming and batch job to the same cluster. We plan to get rid of this restriction for Hybrid Shuffle mode in a future release.**
0 commit comments