Repository navigation
test(e2e): replaced python connection script with bash script in sparkconnect_dependencies_test.go - #3273
Open
udaya-arka wants to merge 12 commits into
Open
test(e2e): replaced python connection script with bash script in sparkconnect_dependencies_test.go#3273udaya-arka wants to merge 12 commits into
sparkconnect_dependencies_test.go#3273udaya-arka wants to merge 12 commits into
Conversation
Add spec.deps to Spark Connect with the Maven-related fields from the review: jars, packages, excludePackages, and repositories. The controller passes them to spark-submit as --jars, --packages, --exclude-packages, and --repositories. - Regenerate the deepcopy, OpenAPI, Swagger, CRD, API docs, and Python API artifacts. - Document .spec.deps in the Spark Connect user guide. - Add examples/sparkconnect/spark-connect-dependencies.yaml. - Add an e2e test that starts the server from the new example and checks that the dependency flags reach spark-submit. - Drop two unnecessary string conversions flagged by golangci-lint. Fixes kubeflow#2959 Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
…add-a-dependencies-fie
- Match examples/sparkconnect/spark-connect.yaml apart from spec.deps so the e2e isolates the dependency behaviour, and use external artifacts that are not already in the base Spark image (re2j jar, spark-avro package, with transitive excludePackages). Drop spec.repositories since Maven Central is already in the runtime and third-party archives add no signal. - Use the Kubeflow Authors boilerplate in both files. - e2e now waits for server and executor pod readiness and runs an actual Spark Connect query that resolves the declared spark-avro package (to_avro), instead of asserting the container args that the unit tests already cover. Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
…ark Connect server subm Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
The dependency e2e drove spark-shell with --master 'local[1]', which selects the Classic session; the .remote() URL is ignored in that mode, so to_avro was evaluated on the client JVM and never reached the server. That produced a failure that looked like a dependency problem but was a client-mode problem, and the same query 'passed' against no server at all. Drive the Connect REPL through a pty instead. That keeps the client a real Spark Connect session, so the query runs against the operator's server and sees the packages declared in spec.deps. The helper also asserts the session type, so a classic fallback fails loudly rather than silently. Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
…add-a-dependencies-fie # Conflicts: # internal/controller/sparkconnect/options_test.go
…add-a-dependencies-fie
Addresses the remaining review points on kubeflow#3120: - Add the Kubeflow license header above the helper's docstring. - Prove both halves of spec.deps resolve on the server session by asserting on values: hex(to_avro(42)) from the declared spark-avro package, and reflect('com.google.re2j.Pattern', ...) from the declared jar. A row count could pass with only one of them present. - Assert AVRO=54 and RE2J=true in the Go test instead of CONNECT_ROWS=10. - Drop the explanatory comment from the example manifest as requested. Verified against a local Spark Connect server started with this example's spec.deps: AVRO=54, RE2J=true. Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
The example manifest began with a bare '#' line before the copyright, which the testing repo's boilerplate does not have. The e2e helper and its Go assertion also carried explanatory comments the reviewer asked to remove. Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
🎉 Welcome to the Kubeflow Spark Operator! 🎉 Thanks for opening your first PR! We're happy to have you as part of our community 🚀 Here's what happens next:
Join the community:
Feel free to ask questions in the comments if you need any help or clarification! |
Contributor
|
@udaya-arka can you please sign off your commits |
…kconnect_dependencies_test.go Signed-off-by: Prabhath Nampally <prabhathn03@gmail.com>
Signed-off-by: Prabhath <14313773+udaya-arka@users.noreply.github.com>
udaya-arka
force-pushed
the
agent/3260-reuse-spark-shell-harness-test
branch
from
October 29, 2026 03:53
b90c338 to
624a6f9
Compare
Author
|
Hi @vikas-saxena02, my bad. |
…query.py Signed-off-by: Prabhath Nampally <prabhathn03@gmail.com>
Author
|
@vikas-saxena02 Thanks for reviewing the PR. I have done the changes to issues pointed out by you. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of this PR
Proposed changes:
connect_query.pyChange Category
Rationale
Checklist
Additional Notes