You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
workspaces: add Spark Connect operator and user guides for notebooks and workspaces - #4479
Adds guides for running Spark from a Kubeflow notebook pod with the Kubeflow SDK, so
users can start a Spark Connect session without hand-writing SparkApplication
manifests. Covers both Notebooks and Workspaces, since Workspaces is still beta and
Notebooks is what most users are running today.
All pages are deliberately short and defer the detail to the integration tests in
kubeflow/community-distribution, which CI keeps green on every pull request. Anything
that would date quickly — the per-session pod settings, the client/server version
pairing — lives there rather than here.
notebooks/spark-connect.md — the permission check, installing the client, a minimal SparkClient.connect() example, and releasing the session. No administrator setup is
documented because none is needed: a notebook pod runs as default-editor, which a
Profile already binds to kubeflow-edit, and the aggregated kubeflow-spark-edit
ClusterRole carries sparkconnects.
workspaces/user-guides/spark-connect.md — the same flow for a Workspace. The only
pod-level setting shown is the driver ServiceAccount, via Driver(service_account=...).
workspaces/operator-guides/spark-connect.md — the one change an administrator must
make for Workspaces, binding kubeflow-spark-edit to the WorkspaceKind's ServiceAccount,
and why Spark pods have to stay out of the service mesh. Workspaces v2.0.0-beta.0 stopped
granting the Kubeflow ClusterRoles to ws-{WORKSPACE_NAME}, which is why this step exists
for Workspaces and not for Notebooks.
workspaces/user-guides/_index.md — new section landing page.
The mesh exclusion is presented as a platform concern in both guides rather than something
a data scientist configures. Its root cause is upstream: SPARK-42411.
Changes since the first revision, from review:
Pages cut by roughly half, with the detail replaced by links to the tested examples
(@juliusvonkohout).
The pod template overrides are gone from the user guides. The driver ServiceAccount now
uses the connect API's own field, and the Istio content moved out of the user-facing
flow (@tariq-hasan).
Both tests the guides link to are merged in kubeflow/community-distribution and share one
session script, so the Notebooks and Workspaces examples are the same code exercised from
two pod types.
vikas-saxena02
changed the title
workspaces: add Spark Connect operator and user guides
notebooks: (actually its workspaces) add Spark Connect operator and user guides
Sep 4, 2026
We do the same for kserve. just the high level and a link to constantly tested automation to reduce the maintenance and prevent it from becoming outdated.
We do the same for kserve. just the high level and a link to constantly tested automation to reduce the maintenance and prevent it from becoming outdated.
@juliusvonkohout can you please share the link to the documentation for kserve and I will align it to the same.
@juliusvonkohout@christian-heusel i ahve raised kubeflow/community-distribution#3631 with an updated example, this code works with sdk v0.4.0, 0.5.0 and the code in main/master branch ... once this is merged, i will update this pr with the link the example added in community distribution repo
@juliusvonkohout@christian-heusel i ahve raised kubeflow/community-distribution#3631 with an updated example, this code works with sdk v0.4.0, 0.5.0 and the code in main/master branch ... once this is merged, i will update this pr with the link the example added in community distribution repo
merged, please update and better just link to the test in the community distribution and reduce this PR here by 50 %. Otherwise this page can get outdated quite fast.
/retitle workspaces: add Spark Connect operator and user guides
google-oss-prowBot
changed the title
notebooks: (actually its workspaces) add Spark Connect operator and user guides
workspaces: add Spark Connect operator and user guides
Oct 15, 2026
@juliusvonkohout@christian-heusel i ahve raised kubeflow/community-distribution#3631 with an updated example, this code works with sdk v0.4.0, 0.5.0 and the code in main/master branch ... once this is merged, i will update this pr with the link the example added in community distribution repo
merged, please update and better just link to the test in the community distribution and reduce this PR here by 50 %. Otherwise this page can get outdated quite fast.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The guides omit the Kubeflow-specific Spark role prerequisite, leaving upstream-chart installations without the permissions needed to follow the examples.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Add Spark aggregated roles prerequisite and qualify permissions claim
The upstream Spark Operator chart does not install kubeflow-spark-edit, so an administrator can satisfy this prerequisite but still have no role to grant to Workspaces. Add the community distribution's Spark aggregated roles as a prerequisite, and qualify the automatic-permissions claim in content/en/docs/components/notebooks/spark-connect.md:11 with the same requirement.
The guides omit the Kubeflow-specific Spark role prerequisite, leaving upstream-chart installations without the permissions needed to follow the examples.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Add Spark aggregated roles prerequisite and qualify permissions claim content/en/docs/components/workspaces/operator-guides/spark-connect.md:20
The upstream Spark Operator chart does not install kubeflow-spark-edit, so an administrator can satisfy this prerequisite but still have no role to grant to Workspaces. Add the community distribution's Spark aggregated roles as a prerequisite, and qualify the automatic-permissions claim in content/en/docs/components/notebooks/spark-connect.md:11 with the same requirement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vikas-saxena02
changed the title
tariq-hasan
left a comment
juliusvonkohout
left a comment
Description of Changes
Adds guides for running Spark from a Kubeflow notebook pod with the Kubeflow SDK, so
users can start a Spark Connect session without hand-writing
SparkApplicationmanifests. Covers both Notebooks and Workspaces, since Workspaces is still beta and
Notebooks is what most users are running today.
All pages are deliberately short and defer the detail to the integration tests in
kubeflow/community-distribution, which CI keeps green on every pull request. Anything
that would date quickly — the per-session pod settings, the client/server version
pairing — lives there rather than here.
notebooks/spark-connect.md— the permission check, installing the client, a minimalSparkClient.connect()example, and releasing the session. No administrator setup isdocumented because none is needed: a notebook pod runs as
default-editor, which aProfile already binds to
kubeflow-edit, and the aggregatedkubeflow-spark-editClusterRole carries
sparkconnects.workspaces/user-guides/spark-connect.md— the same flow for a Workspace. The onlypod-level setting shown is the driver ServiceAccount, via
Driver(service_account=...).workspaces/operator-guides/spark-connect.md— the one change an administrator mustmake for Workspaces, binding
kubeflow-spark-editto the WorkspaceKind's ServiceAccount,and why Spark pods have to stay out of the service mesh. Workspaces v2.0.0-beta.0 stopped
granting the Kubeflow ClusterRoles to
ws-{WORKSPACE_NAME}, which is why this step existsfor Workspaces and not for Notebooks.
workspaces/user-guides/_index.md— new section landing page.The mesh exclusion is presented as a platform concern in both guides rather than something
a data scientist configures. Its root cause is upstream:
SPARK-42411.
Changes since the first revision, from review:
(@juliusvonkohout).
uses the connect API's own field, and the Istio content moved out of the user-facing
flow (@tariq-hasan).
in the community-distribution test scripts, and chore(spark): replace pyspark-connect with pyspark-client (#781) sdk#795 is changing the
underlying package anyway.
Both tests the guides link to are merged in kubeflow/community-distribution and share one
session script, so the Notebooks and Workspaces examples are the same code exercised from
two pod types.
Related Issues
Related: kubeflow/notebooks#1362
Checklist