Add bearer token auth support for OpenSearch clusters in Elasticsearch source#5125
Open
anon767 wants to merge 6 commits into
Open
Add bearer token auth support for OpenSearch clusters in Elasticsearch source#5125anon767 wants to merge 6 commits into
anon767 wants to merge 6 commits into
Conversation
…h source
OpenSearch clusters aren't fully compatible with the existing Elasticsearch
source despite sharing an API surface, so scanning them previously failed:
- JWT auth: OpenSearch's JWT auth plugin expects a raw
"Authorization: Bearer <jwt>" header, which none of the existing auth
options (basic, API key, service token) produce. A new --bearer-token
flag / ELASTICSEARCH_BEARER_TOKEN env var sets this directly.
- Product header: the vendored go-elasticsearch client rejects any
response missing the X-Elastic-Product header, which OpenSearch doesn't
reliably send. A custom http.RoundTripper injects it after the fact when
bearer-token auth is used (i.e. only for the OpenSearch path).
- Point-in-time API: OpenSearch exposes PIT creation at
POST /{index}/_search/point_in_time rather than Elasticsearch's
POST /{index}/_pit, so the vendored esapi.OpenPointInTimeRequest 404s;
added an OpenSearch-specific PIT-open call for this path.
- Sort tiebreaker: Elasticsearch's PIT tiebreaker field is "_shard_doc";
OpenSearch doesn't recognize it and rejects the search with a
query_shard_exception, but accepts the equivalent "_doc".
All four are gated behind bearer-token being set, so existing Elasticsearch
behavior (basic/API-key/service-token auth) is unchanged.
Found via live testing against a real OpenSearch 2.17 cluster with JWT
auth: when no --query-json/--since-timestamp/resumed timestamp is set,
the search body's "query" field was an empty {} object. Elasticsearch
tolerates this as an implicit match-all, but OpenSearch's stricter query
parser rejects it outright with "illegal_argument_exception: query
malformed, empty clause found", silently zeroing out every scan.
match_all is the explicit, portable equivalent and works identically on
both, so this isn't gated behind isOpenSearch.
|
|
Contributor
|
Can you add some tests that demonstrate the current Elasticsearch source doesn't work with OpenSearch, but that they pass with this PR? I think that would clearly show the value of the change and help ensure we don't regress in the future. |
TestSource_OpenSearchAPI runs a real opensearchproject/opensearch:2.17.0 container and calls processSearchedDocuments directly against it, showing both sides of the fix: isOpenSearch=false fails, isOpenSearch=true succeeds. Verified locally: passes, and pre-existing TestSource_ElasticAPI (real Elasticsearch 8.9.0) still passes unmodified.
Author
|
Done |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5124
Summary
OpenSearch clusters share TruffleHog's Elasticsearch API surface but aren't fully compatible with the existing
elasticsearchsource, so scanning an OpenSearch cluster currently fails. This adds a--bearer-tokenflag (ELASTICSEARCH_BEARER_TOKENenv var) that works around five incompatibilities (see linked issue), all gated behind bearer-token auth being set so existing Elasticsearch behavior (basic/API-key/service-token) is unchanged.Test plan
go build ./...passesisOpenSearchparameter (passedfalse, no behavior change)--bearer-tokenauthenticates, PIT opens via the OpenSearch-specific endpoint, the_doctiebreaker sort works, and documents are correctly fetched and chunked for scanning. Seeded a document containing a JWT-shaped test secret and confirmed TruffleHog's own JWT detector correctly fired on it end-to-end through this path (source: Elasticsearch, index test-logs) -- this testing is also what surfaced the empty-query-clause bug above.Notes for reviewers
Nothing about existing Elasticsearch source behavior changes -- the auth/PIT/tiebreaker workarounds only activate when
bearer_tokenis non-empty; the query-clause fix is a strict improvement on both backends.Note
Low Risk
Small, additive credential wiring with no change to default Elasticsearch auth when bearer token is omitted; secrets still follow the same flag/env patterns as other cluster credentials.
Overview
Adds
--bearer-token(andELASTICSEARCH_BEARER_TOKEN) to theelasticsearchscan command so OpenSearch clusters that use JWT/Bearer auth can authenticate the same way as other Elasticsearch credential flags.The value is passed through
sources.ElasticsearchConfiginto the scan engine’s Elasticsearch protobuf connection, and the man page documents the new option. Existing username/password, API key, and service-token flows are unchanged when bearer token is not set.Reviewed by Cursor Bugbot for commit 07fabde. Bugbot is set up for automated code reviews on this repo. Configure here.