[vikas-saxena02]

Purpose of this PR

Fixes #3112.

spec.server.cores was passed to the Connect server as spark.driver.cores, which has no effect in client mode, so the field did nothing. As agreed on the issue, cores is kept and now acts as a shorthand for spec.server.coreRequest (added in #3096).

Proposed changes:

• Stop passing spark.driver.cores to the Connect server.
• Set the server pod's CPU request from server.cores when server.coreRequest is unset. coreRequest still takes precedence.
• Apply the same precedence in the webhook's effective request/limit check.
• Update the Cores field description and regenerate the API artifacts, the user guide and the example.

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

See the reproduction and discussion on #3112. Executor behaviour is unchanged: executor.cores still maps to spark.executor.cores.

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

For reviewers:

1. Precedence over the pod template: server.cores now overrides a CPU request set in server.template, the same way coreRequest does. A spec with cores: 1 and a template CPU limit below 1 was accepted before, because cores had no effect, and is now rejected by the webhook. Is that acceptable, or should cores fall back below the template instead?
2. Request only: cores sets only the CPU request, not the limit, per the issue discussion. The driver's availableProcessors() follows the CPU limit, so JVM parallelism is unchanged unless coreLimit is also set.
3. server.memory: left unchanged in this PR. It still sizes the heap via spark.driver.memory but not the pod. Happy to follow up separately if wanted.

---

AI assistance disclosure: The documentation changes and this PR description were drafted with help from Claude (Anthropic) and reviewed by me. The unit tests had local failures those were remediated with help from Claude as well.

[vikas-saxena02]

@nabuskey @tariq-hasan can you plz review this PR for me. I will attend to ci-failures (if any) tomorrow morning, off to bed now

[vikas-saxena02]

@nabuskey @vjanelle @tariq-hasan can i please get a review on this one as well

[vikas-saxena02]

/retest

[vikas-saxena02]

/retest

[nabuskey]

/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nabuskey

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [nabuskey]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment