Conversation
The rate was wrong as it was applies a LIMIT twice
This reverts commit e2c741e.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1234 +/- ##
===========================================
+ Coverage 98.45% 100.00% +1.54%
===========================================
Files 18 9 -9
Lines 1425 603 -822
Branches 48 19 -29
===========================================
- Hits 1403 603 -800
+ Misses 22 0 -22
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
A uniform draw from anomaly='t' oversamples whichever mechanism dominates globally, in practice TLS resets, so the corpus calibrates one layer and starves the others. Three new mutually exclusive strata on analysis_web_measurement (screen_dns, screen_tcp, screen_tls) attribute positives to the first blocked-leaning layer, and a `shares` query parameter steers how the queue splits across strata. Shares only move analyst effort: row weights stay population/drawn, so no share choice can bias an estimate, only its variance. Like `limit`, shares are not part of the design fingerprint; the same design with a bigger share returns a superset of the same rows. screen_positive now excludes confirmed='t'. Confirmed rows are certain blocking with a known artefact, so labelling one adds almost nothing, and they were eating the positive quota; they keep their own fingerprint_match census. The layer strata read the pipeline's own scores, so they oversample what the pipeline already sees; screen_negative remains the only stratum that can discover what it misses, which is why its share should never be cut. analysis_web_measurement queries now use FINAL: during a reprocess the same uid exists in old and new versions until the merge runs, which inflated populations and could draw a uid twice. DESIGN_SCHEMA_VERSION bumped to 2 so old design ids are never reinterpreted under the new strata.
The threshold that turns a continuous *_blocked score into a verdict was written out three times: `>= 0.5` twice in the labeling router and `> 0.5` in the aggregation router. Those disagree at exactly 0.5, and that is not a hypothetical boundary — dns.failure_no_ctrl (DNS failing with no usable control) scores exactly 0.5 and is the only rule that does. Measured against a day of production: 7,970 measurements land exactly on it, 3.2% of everything flagged, all of them blocked to the labeller and not blocked to the aggregation API. scoring.py now holds BLOCKING_THRESHOLD and generates the predicates, so the sampling strata and the aggregation query cannot drift apart again. The comparison is >=, which is what the labelling corpus was drawn under, so any calibration fitted on it describes deployed behaviour. This changes the aggregation API: ~3% more measurements appear in likely_blocked_protocols. attributed_to() replaces the hand-written layer predicates. Layers gate each other, so attribution requires every earlier layer to be quiet; a test over a grid that straddles the boundary asserts the three are mutually exclusive and partition exactly what any_blocked() selects, which is what sampling weights built from them depend on. Threshold arguments resolve the module global at call time rather than binding it as a default, so a test or config layer that overrides the constant is not silently ignored. Responses and sampling designs now carry scoring_version and the threshold. A verdict is only interpretable next to the regime that produced it, and the layer strata are *defined* by the threshold, so labels drawn either side of a change are not one population and must not pool.
The fuzzy score is built from hand-set constants: it ranks measurements sensibly but its magnitude is not a probability, and ontology.md warns that shipping it as though it were is exactly how it gets misread. Map it through a logistic fitted against the adjudicated corpus instead, so the published number is falsifiable — group everything reported at 0.9 and about 90% of it should be blocked. /v1/analysis gains blocked_probability per measurement. The aggregation gains blocked_probability_mean, the mean of the per-measurement probabilities over the cell, which estimates the share of it that is blocked. Deliberately not the probability of the aggregated score: that would answer "is the 99th-percentile measurement blocked", which nobody asks. Neither is stored, so a recalibration is a deploy rather than a reprocess. Three things the fit needs that a default LogisticRegression call would get wrong, all recorded in Calibration: - Sampling weights. The corpus is 30.3% positive against a population rate of 10.0%, so an unweighted fit describes the corpus and overstates blocking by roughly that ratio. - A ridge penalty on the slope. The classes are nearly separable at this corpus size; unpenalised, the bootstrap interval on the slope runs to 6521 and leave-one-out log loss blows up to 8.2 against 0.025 here. The penalty is selected by leave-one-out weighted log loss, not chosen. - The interval, and the range over which the corpus actually constrains the curve. The tails are extrapolation from a handful of points, so 0.995 and 0.95 are not different claims and the response says so. log10_odds_to_prob is named for its base because the failure mode is silent: 1/(1+exp(-x)) on a log10 score agrees at 0, stays inside [0,1] everywhere, and is simply wrong in between. The SQL uses 1/(1+pow(10,-x)) rather than pow(10,x)/(1+pow(10,x)), which returns nan at the top of the range — verified against the deployed ClickHouse. The fitted threshold-equivalent is P=0.506 at the deployed 0.5, so publishing probabilities changes no decisions. A test pins that, since a refit breaking it means the two knobs have drifted and one needs revisiting deliberately.
The detector harness prints "false alerts per quiet series-week", but nothing anywhere defined a quiet series-week. The event corpus cannot: it is curated, so recall over it is a coverage statement about a hand-built set, and a rate needs quiet time counted rather than collected. Quiet time can only be counted if it was sampled from a frame, which is what this adds. /interval_sample draws (probe_cc, probe_asn, domain) x ISO week — the detector's own cell, keyed exactly as event_detector_cusums is, because anything coarser estimates a rate over a different population than the one the detector runs on. Each row carries the population it was drawn from and the predicate that defined it, so the weights are reconstructable from an export alone, as they are for measurements. The strata partition the frame rather than overlapping. Drawing weeks the incumbent alerted in and, separately, random covered weeks gives an alerted cell-week two selection probabilities and no correct weight, so random_covered is resolved as the complement of whatever else is being drawn and the resolved predicate goes into the design spec. That keeps the alerted stratum from being circular — on its own it estimates precision given firing, not a rate over quiet time — while a third, optional near_miss stratum importance-samples the weeks that scored blocked-leaning without alerting, where the disagreements live. Three properties of the frame are recorded because each is a way to flatter a detector for free: a volume floor, since a uniform draw is dominated by cells too thin for anything to fire in; whole ISO weeks, since a partial week is a shorter observation window rather than a smaller one; and the domain set the detector actually runs on, since quiet time in cells nothing watches is not evidence. All three are in the design id. /interval_reveal is the post-commit half. Blinding matters more on this grain than on measurements: one stratum is the detector's output, so an unblinded alert state does not merely anchor the analyst, it hands them the answer. Nothing in the candidate path returns a changepoint, a CUSUM state or a score.
Must not be run while a backfill is in progress
Documentation and one comment; no behaviour change. The sampler's docstrings described a single silent verdict, but the labeller now distinguishes a week where OONI saw nothing wrong from a week inside a block that started earlier. Both mean no transition happened, so both are weeks the detector should have been silent through and both belong in the false-alarm denominator — "silent series-week" rather than "quiet series-week", which is the phrase the evaluation notebook now prints. Also records why the reveal's signal query does not use FINAL while the frame query does: it is read to draw a chart rather than to size a population, so an unmerged duplicate nudges a median instead of moving a cell-week into another volume band and corrupting a weight.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.