Skip to content

[CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures #3877

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
afterincomparableyum wants to merge 2 commits into from
Closed

[CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures #3877

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a69e2a4
[CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Mave…
afterincomparableyum Oct 17, 2026
4ebbfad
[CELEBORN-2484][BUILD] Use curl --output so retries discard partial d…
afterincomparableyum Oct 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion build/sbt-config/repositories
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
# fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
# assigned within the build/sbt-launch-lib.bash script.
private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/repos/central/data/
gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/
maven-central
typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
Expand Down
35 changes: 25 additions & 10 deletions build/sbt-launch-lib.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,13 @@ acquire_sbt_jar () {
if [ -n "$DEFAULT_ARTIFACT_REPOSITORY" ]; then
export SBT_OPTS="$SBT_OPTS -Dceleborn.sbt.default.artifact.repository=${DEFAULT_ARTIFACT_REPOSITORY}"
fi
URL1=${DEFAULT_ARTIFACT_REPOSITORY:-https://repo1.maven.org/maven2/}org/scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch-${SBT_VERSION}.jar
SBT_LAUNCH_PATH=org/scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch-${SBT_VERSION}.jar
if [ -n "$DEFAULT_ARTIFACT_REPOSITORY" ]; then
URLS="${DEFAULT_ARTIFACT_REPOSITORY}${SBT_LAUNCH_PATH}"
else
# Fall back to Google's Maven Central mirror when repo1 is unreachable or rate limited.
URLS="https://repo1.maven.org/maven2/${SBT_LAUNCH_PATH} https://maven-central.storage-download.googleapis.com/maven2/${SBT_LAUNCH_PATH}"
fi
JAR=build/sbt-launch-${SBT_VERSION}.jar

sbt_jar=$JAR
Expand All @@ -73,16 +79,25 @@ acquire_sbt_jar () {
# Download
printf "Attempting to fetch sbt\n"
JAR_DL="${JAR}.part"
if [ $(command -v curl) ]; then
curl --fail --location --silent ${URL1} > "${JAR_DL}" &&\
mv "${JAR_DL}" "${JAR}"
elif [ $(command -v wget) ]; then
wget --quiet ${URL1} -O "${JAR_DL}" &&\
mv "${JAR_DL}" "${JAR}"
else
printf "You do not have curl or wget installed, please install sbt manually from https://www.scala-sbt.org/\n"
exit -1
CURL_RETRY_OPTS="--retry 3 --retry-delay 5"
# --retry alone skips DNS, connection and 403 errors; --retry-all-errors needs curl 7.71+.
if [ $(command -v curl) ] && curl --help all 2>/dev/null | grep -q -- "--retry-all-errors"; then
CURL_RETRY_OPTS="${CURL_RETRY_OPTS} --retry-all-errors"
fi
for URL in ${URLS}; do
if [ $(command -v curl) ]; then
curl --fail --location --silent --show-error ${CURL_RETRY_OPTS} --output "${JAR_DL}" ${URL} &&\
mv "${JAR_DL}" "${JAR}"
elif [ $(command -v wget) ]; then
wget --quiet --tries=4 --waitretry=5 ${URL} -O "${JAR_DL}" &&\
mv "${JAR_DL}" "${JAR}"
else
printf "You do not have curl or wget installed, please install sbt manually from https://www.scala-sbt.org/\n"
exit -1
fi
[ -f "${JAR}" ] && break
printf "Failed to fetch sbt from ${URL}\n"
done
fi
if [ ! -f "${JAR}" ]; then
# We failed to download
Expand Down
Loading