[CELEBORN-2474] Send WWW-Authenticate challenge when no auth header present

[senthh]

What changes were proposed in this pull request?

When AuthenticationFilter.doFilter receives a request with no Authorization header (the first request in a SPNEGO/Kerberos handshake), respond 401 with WWW-Authenticate headers listing all registered auth schemes, so a SPNEGO client (e.g. curl --negotiate) can receive the challenge and respond with its Kerberos token. If an Authorization header is present but matches no handler, the original error message is preserved.

Also fix the scheme name in the supportedSchemes documentation (celeborn.master.http.auth.supportedSchemes / celeborn.worker.http.auth.supportedSchemes): the Kerberos scheme was listed as SPNEGO but the enumeration value is NEGOTIATE. Updated CelebornConf.scala, docs/configuration/master.md and docs/configuration/worker.md for consistency with the ConfigurationSuite golden-file check.

Files:

• service/src/main/scala/org/apache/celeborn/server/common/http/authentication/AuthenticationFilter.scala
• common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
• docs/configuration/master.md
• docs/configuration/worker.md

Why are the changes needed?

Previously the filter returned a bare 401 with no WWW-Authenticate header, so the SPNEGO handshake was broken at the very first step (error: "No auth scheme matched for null"). A SPNEGO client never receives the challenge and cannot send its Kerberos token.

Does this PR resolve a correctness bug?

• Yes

Does this PR introduce any user-facing change?

• Yes

How was this patch tested?

Verified with curl --negotiate against a Celeborn server configured with:

• celeborn.master.http.auth.supportedSchemes NEGOTIATE
• celeborn.master.http.spnego.keytab /path/to/keytab
• celeborn.master.http.spnego.principal HTTP/host@REALM

Before: 401 without WWW-Authenticate header, curl never sends the token.
After: 401 with WWW-Authenticate: NEGOTIATE, curl completes the SPNEGO handshake and returns 200.

[RexXiong]

Could we include a realm in the Basic challenge, e.g. Basic realm="Celeborn", instead of emitting only the scheme name for every handler?

RFC 7617 requires a realm for Basic authentication. I verified with Python’s HTTPBasicAuthHandler that a bare WWW-Authenticate: BASIC results in a 401 without retrying with credentials, while adding the realm allows authentication to proceed.

The existing BasicAuthenticationHandler.authenticate() path also sets a bare Basic challenge when the decoded credentials are incomplete. Could we update that response to include the same realm as well, so both paths return a consistent challenge? This is pre-existing behavior, but it would be useful to fix it alongside the new challenge logic.

It would also be helpful to add regression tests for missing/empty Authorization headers and multiple configured schemes.

AI-assisted review by OpenAI Codex (GPT-6).

[Copilot]

🟡 Changes recommended

The core unauthenticated challenge behavior lacks automated regression coverage.

1 open finding

• Add test for unauthenticated challenges across all authentication schemes · New

What changed in this PR

Adds authentication challenges for unauthenticated HTTP requests, enabling SPNEGO clients to initiate Kerberos negotiation.

Changes:

• Returns all configured WWW-Authenticate schemes with 401 responses.
• Corrects SPNEGO configuration terminology to NEGOTIATE.

File	Description
AuthenticationFilter.scala	Adds authentication challenges for missing headers.
CelebornConf.scala	Corrects supported-scheme documentation.
docs/​configuration/​master.md	Updates generated master configuration docs.
docs/​configuration/​worker.md	Updates generated worker configuration docs.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.