[fix][broker][proxy] Add httpMaxResponseHeaderSize config and preemptive header overflow check

[geniusjoe]

Fixes #26229

Motivation

When a message contains properties that cause the HTTP response header size to exceed Jetty's HttpConfiguration.responseHeaderSize (default 8KB), the Admin API endpoints getMessageById/getMessagesById, peekNthMessage, and examineMessage fail silently. Since message metadata is placed in response headers like X-Pulsar-PROPERTY-*, the 8KB default limit is relatively easy to hit — for example, a message with a few dozen properties or several long property values can exceed it.

When the header size exceeds the limit, Jetty cannot write any valid HTTP response and directly resets/closes the connection. This causes two problems:

1. Server side: The error is logged to stdout (via java.util.logging) instead of pulsar.log, because Jersey catches the BadMessageException: 500: Response header too large in ServerRuntime$Responder.writeResponse and logs it at SEVERE level via JUL, which goes to stdout. This makes the issue hard to diagnose.
2. Client side: The caller receives a generic PulsarAdminException with statusCode=500 wrapping a TimeoutException, with no indication that the root cause is header overflow. The 500 status code is misleading — it does not mean the server returned HTTP 500; it is the default error code assigned client-side when no HTTP response was received.

See the detailed root cause analysis in #26229 for the full call chain.

Modifications

• Add httpMaxResponseHeaderSize configuration to both ServiceConfiguration (broker) and ProxyConfiguration (proxy), with a default value of 8192 bytes (matching Jetty's default). Apply the config to Jetty's HttpConfiguration.responseHeaderSize in WebService (broker) and WebServer (proxy).
• Add a preemptive header size estimation check in PersistentTopicsBase.generateResponseWithEntry(), which is the shared response generation path for getMessageById/getMessagesById, peekNthMessage, and examineMessage:
  • Estimate the total response header size by iterating over Response.getStringHeaders().
  • When the estimated header size exceeds httpMaxResponseHeaderSize, throw a RestException with HTTP 431 (Request Header Fields Too Large) and a descriptive error message that includes both the estimated size and the configured limit, and suggests increasing httpMaxResponseHeaderSize.
  • Log the event at ERROR level to pulsar.log with structured attributes (topic, ledgerId, entryId, estimatedHeaderSize, httpMaxResponseHeaderSize).
• Fix RestException propagation in callback handlers to prevent 431 being wrapped as 500:
  • internalGetMessageById (readEntryComplete): Catch RestException separately before catch (Exception).
  • internalPeekNthMessageAsync: Add catch (RestException) to the lambda.
  • internalExamineMessageAsync (generateResponseWithEntry call): Add catch (RestException) before catch (IOException).
• Refactor generateResponseWithEntry to build headers into a headerOnlyResponse first, estimate the size, then construct the final response with Response.fromResponse(headerOnlyResponse).entity(stream).build().
• Add broker.conf and proxy.conf entries for the new configuration.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• testGetMessageByIdHeaderTooLarge431: Verifies that getMessageById returns HTTP 431 with a descriptive error message containing "exceeds HTTP response header limit" and "httpMaxResponseHeaderSize" when the response header exceeds the configured limit.
• testGetMessageByIdHeaderWithinLimit: Verifies that getMessageById works normally when the response header is within the default 8KB limit (50 properties).

Does this pull request potentially affect one of the following parts:

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
  • A new configuration httpMaxResponseHeaderSize is added to both broker and proxy with a default value of 8192 bytes, which matches Jetty's existing implicit responseHeaderSize default. This does not change any existing default behavior.
• The threading model
• The binary protocol
• The REST endpoints
  • getMessageById/getMessagesById, peekNthMessage, examineMessage now return 431 instead of 500 when response header size exceeds the limit
• The admin CLI options
• The metrics
• Anything that affects deployment