Repository navigation
feat: Spec getExperimentFlag under local evaluation - #21
Open
gagantrivedi wants to merge 2 commits into
Open
gagantrivedi wants to merge 2 commits into
gagantrivedi wants to merge 2 commits into
Conversation
Add a client step creating a client with both local evaluation and events, which the runner asks for when a scenario needs both; until now it dropped local evaluation. Let a running experiment enroll a given percentage of identities, so that a scenario can run one enrolling none.
gagantrivedi
marked this pull request as ready for review
November 5, 2026 08:42
gagantrivedi
requested review from
khvn26 and
kyle-ssg
and removed request for
a team
November 5, 2026 08:42
Zaimwa9
requested review from
Zaimwa9
and removed request for
khvn26 and
kyle-ssg
November 5, 2026 08:53
Zaimwa9
requested changes
Nov 5, 2026
| "experiment_rollout": { | ||
| "enabled": True, | ||
| "rollout_percentage": 100, | ||
| "rollout_percentage": rollout_percentage, |
Contributor
There was a problem hiding this comment.
Below L144 we hardcode percentage_allocation so it means no enrolled identities will evaluate to control . Could we add a scenario with 0% allocation to the treatment and verify we get an exposure with control?
Member
Author
There was a problem hiding this comment.
Added in cb589d7: the experiment step now takes the variant's share of enrolled identities, and a new scenario enrols an identity at 0% treatment and expects a "control" exposure.
| | checkout | true | "new checkout" | | ||
| And Flagsmith has received the following events: | ||
| | event | feature | identifier | value | traits | | ||
| | $flag_exposure | checkout | user_123 | "treatment" | {"plan": "premium"} | |
Contributor
There was a problem hiding this comment.
It would be worth to mention metadata for experiment_id now so it's on the agent radar and makes it easier to be on par with OpenFeature.
Although we parked the status but metadata.experiment_id is only present when the experiment is running so that would cover the paused / completed states
gagantrivedi
added a commit
to Flagsmith/flagsmith-go-client
that referenced
this pull request
Nov 6, 2026
Since Flagsmith/flagsmith#8532, the environment document carries `metadata.experiment` on feature states. The SDK dropped it when mapping the document for the engine, so under local evaluation `Flag.Experiment` was always nil and `GetExperimentFlag` never recorded an exposure. ### Changes - `FeatureStateModel` reads `metadata.experiment`; the engine's `FeatureMetadata` gains `Experiment`. - The mapper copies it for environment and segment feature states. Identity overrides carry none, as on the API. - Flags built from evaluation results set `Experiment`. - Locally evaluated environment flags carry no experiment, matching the API, which only sends it for identity flags. - Bump `engine-test-data` to v3.13.0 for its experiment cases (Flagsmith/engine-test-data#64). - README and doc comments no longer say exposures need remote evaluation. ### Tests `GetExperimentFlag` under local evaluation: an enrolled identity records one exposure with its variant and experiment ID; an identity outside the rollout, one with an identity override, and a feature with no experiment record none. Environment flags carry no experiment. Also ran Flagsmith/sdk-spec#21 against this branch: 12 passed.
khvn26
reviewed
Nov 6, 2026
| feature: str, | ||
| variant: str, | ||
| value: str, | ||
| rollout: str | None, |
Member
There was a problem hiding this comment.
nit
Suggested change
| rollout: str | None, | |
| rollout: int, |
maybe it's better to supply explicit rollout value in step definitions now that we have cases with varying rollouts
An experiment now states both its rollout and the share of enrolled identities served the variant, so a scenario can enrol an identity into control. Event expectations can match the scenario's experiment id with "<experiment>".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Specs that
getExperimentFlagrecords exposures under local evaluation, as it does under remote evaluation. Since Flagsmith/flagsmith#8532 the environment document carries each feature state's experiment, so a server-side SDK can tell whether an identity is enrolled without asking Flagsmith.Changes
features/local__get_experiment_flag.feature(@local-evaluation @events): an enrolled identity is exposed to its variant; an identity a 0% rollout does not enroll, and one outside any experiment, are not.Create the SDK client with local evaluation and events enabled, taking the same arguments asCreate the SDK client with events enabled. The runner asks for it when a scenario needs both; until now it silently dropped local evaluation.a running experiment on feature … with value …takes an optionalto N% of identities, defaulting to every identity as before.Identity overrides aren't covered: self-hosted Flagsmith doesn't put them in the environment document.
Verified
Ran the whole spec against the Python SDK with Flagsmith/flagsmith-python-client#276 and the new client step: 12 passed. Against the SDK's
main, the enrolled scenario fails with no exposure delivered.