Repository navigation
docs: add perceptual evaluation methodology for splats - #35
Conversation
PSNR, SSIM and LPIPS are the only signals the pipeline uses to compare splat configurations. Three subjective studies put all three near 0.5 Spearman correlation with human opinion scores, which is not enough to settle the cap_max, cleanup threshold and container decisions we have been settling with them. Collects the numbers from 3DGS-VBench, MUGSQA, 3DGS-QA and GS-QA, names which of our conclusions they weaken, and proposes running DOVER on the orbit renders already on disk as the cheapest way to check. Refs #34
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f45b95ce55
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 1. Run DOVER on the orbit renders from the cap_max sweep already on disk. Four | ||
| configurations, one scene, no new training. If DOVER separates them where | ||
| PSNR and SSIM did not, the metric earns its place. |
There was a problem hiding this comment.
When this acceptance criterion is followed, any DOVER score difference—even one unrelated or inversely related to perceived quality—causes the metric to “earn its place.” This repeats the document's criticism of LPIPS: separation demonstrates sensitivity, not agreement with viewers. The experiment needs human preferences or stimuli with a known perceptual ordering before it can validate DOVER.
Useful? React with 👍 / 👎.
| stimuli the way a human would about (1+t)/2 of the time. On 3DGS-VBench that | ||
| puts PSNR at 68%, LPIPS at 68% and DISTS at 76%. On 3DGS-QA, LPIPS drops to 64%. | ||
| A coin flip is 50%. When we compare two configurations and one wins on LPIPS, | ||
| the odds that a viewer agrees are closer to two in three than to a settled |
There was a problem hiding this comment.
For Kendall tau computed against aggregate MOS, (1+t)/2 is the proportion of stimulus pairs whose objective ordering agrees with the aggregate MOS ordering; it is not the probability that an individual viewer agrees with the metric on a particular configuration comparison. Consequently, these values do not support the stated “two in three” odds without subject-level preference analysis.
Useful? React with 👍 / 👎.
| The correlations above are pooled across content. Predicting an absolute opinion | ||
| score across 15 object types is harder than ranking two settings within one | ||
| scene, so 0.5 is a lower bound on how much of our problem the metrics miss, not | ||
| a direct estimate of how often our A/B comparisons are wrong. |
There was a problem hiding this comment.
If pooled cross-content prediction is harder than ranking settings within one scene, the pooled correlation may underestimate the metrics' within-scene performance, so the amount they miss in this project's A/B comparisons may be smaller, not bounded below by 0.5. Because correlation is not itself an amount missed, the safest correction is to remove the bound and state only that the pooled value may not transfer directly.
Useful? React with 👍 / 👎.
Two of them were the error this document is about, made inside the document. The DOVER acceptance criterion said the metric earns its place if it separates configurations PSNR and SSIM could not, which is sensitivity, not agreement, and LPIPS already has sensitivity. It now runs a ladder with an ordering nobody disputes, then checks the metric against forced-choice judgments on the pairs we actually cannot settle, and says what to do when it disagrees. The Kendall conversion described agreement with an individual viewer. (1+t)/2 is agreement with the averaged panel; per-viewer agreement is lower and none of these papers reports it. The cross-content caveat had its inequality backwards: if pooling across object types is the harder problem, the pooled figure overstates what the metrics miss within one scene, not understates it. Both directions are plausible here and neither is measured, so it now says that.
|
@codex review The experiment plan in |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e879b22529
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 3. **Adopt or fall back.** If DOVER agrees with the forced choices, wire it in | ||
| after export and report it alongside `ns-eval`. If it disagrees, DISTS (0.73) |
There was a problem hiding this comment.
For the close pairs this experiment is intended to resolve, one rater's forced choice produces a label even when the variants are perceptually indistinguishable, and the three listed binary comparisons can all match DOVER by chance with probability 1/8 under independent random choices. Step 3 then adopts the metric using the same tiny set on which it was selected, so agreement does not validate DOVER for future tuning decisions and disagreement does not show that no metric can resolve them. Allow indifference and repeated/multiple judgments, then apply a predeclared acceptance threshold on a larger independent validation set before wiring the metric in.
Useful? React with 👍 / 👎.
Three binary comparisons agree with any metric one time in eight, and step 3 adopted the metric on the same three that selected it. Both were real holes. The cleanup threshold supplies the pairs for free: 15 scenes, each with an uncropped and a filtered export, cut ranging from 4.9% to 62%. The comparison now offers indifference as a third response, repeats every pair to measure rater self-consistency, and splits selection from validation with the threshold fixed in advance. Names the indifference outcome as a result rather than a failure. If two variants are indistinguishable on an orbit, that settles the decision without any metric.
106 against 212 training views is a tie on all three metrics, with LPIPS moving the wrong way by a twentieth of the per-image spread. The 424 run timed out in COLMAP and is not worth rerunning given the null at 2x.
|
@codex review Both holes were real. Step 2 now draws its pairs from all 15 Gaudi scenes (each has an uncropped and a filtered export, cut ranging 4.9% to 62%) rather than three hand-picked comparisons, offers indifference as a third response, and repeats every pair to measure rater self-consistency. Step 3 splits selection from validation with the threshold fixed in advance. Also recorded the frame-count result, which came back while this was open. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10457b1ced
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| pinned 11-frame held-out set, and came back a tie: PSNR 20.05 against 20.10, | ||
| SSIM 0.519 against 0.532, LPIPS 0.3645 against 0.3677, all inside a per-image | ||
| spread of 0.063 to 0.070. Doubling the training views moved nothing, and moved |
There was a problem hiding this comment.
Because both configurations were evaluated on the same 11 frames, the raw variation among per-image scores does not establish uncertainty in the difference: scene-dependent variation can cancel almost entirely in the paired deltas. A consistently signed improvement could therefore be hidden inside a much larger raw score spread. Report the distribution or confidence interval of the 11 paired differences—or a predeclared practical-equivalence bound—before calling this result a tie.
Useful? React with 👍 / 👎.
| selection set to pick among DOVER, DISTS (0.73) and CW-SSIM (0.74), then | ||
| score the winner once on the held-out set. Count only pairs where the rater |
There was a problem hiding this comment.
DISTS and CW-SSIM are full-reference image metrics, unlike the no-reference video metric DOVER, so an A/B pair of orbit videos alone cannot produce a preferred variant: comparing A directly with B yields only a symmetric similarity/distance, while treating the unfiltered export as the reference predetermines that it wins. Specify aligned ground-truth views and have the human assessment cover those same stimuli, or restrict this selection to metrics that can independently score each displayed orbit.
Useful? React with 👍 / 👎.
Both configurations in the frame-count run were scored on the same 11 frames, so the comparison is paired and the per-image spread is the wrong yardstick. Frame difficulty is shared between the runs and cancels in the per-frame differences, so a consistent 0.003 could hide inside a 0.07 spread. ns-eval writes only aggregates, so the values that would settle it do not exist yet. The same reasoning produced every other "inside the spread" verdict in this project. The error is one-sided: nulls become less safe, findings already called real stay real. The metric candidates also have to be no-reference. An A/B of two orbits has no ground truth, so DISTS and CW-SSIM cannot score it: symmetric distance gives no direction, and using the unfiltered export as the reference hands it the win. They belong to a held-out-frame experiment against different stimuli.
Re-ran eval with per-frame output and tested the differences directly. Opacity pruning at 0.05 is not free: 0.0007 LPIPS on gaudi, 0.0059 on r2d2, same sign on every frame of both, PSNR and SSIM agreeing. The marginal spread is 70x the gaudi effect, which is why it read as noise. PSNR and SSIM being blind to the Gaussian budget survives pairing and gets stronger, with the interval containing zero and the sign splitting 3/5 while LPIPS separates every step. The frame-count experiment turns out to be invalid rather than a tie. The split fraction was validated at full precision and passed as a six-decimal rounding, so ceil returned 107 instead of 106 and both configurations held out ten frames at different positions. The two runs share one ground-truth image out of ten. On FID and KID: the premise needs correcting first, since LPIPS is among the worst metrics in all three studies rather than one that works. FID and KID need no correspondence between the sets, which is the one thing that fits an orbit A/B, but FID is unusable at our sample sizes and its bias depends on the model. KID and CMMD are the versions that survive small n. All three are blind to geometric error and would be scored against capture frames containing the transients we remove.
|
@codex review Substantial new content since the last pass: a section reporting paired per-frame results (which overturned the "opacity pruning is free" verdict and invalidated the frame-count experiment), and a section on FID/KID/CMMD. Both are new claims that have not been reviewed. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: effb5a7272
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| **Opacity pruning is not free.** At 0.05 it costs 0.0007 LPIPS on gaudi and | ||
| 0.0059 on r2d2, and the sign holds on every frame of both scenes, with PSNR and | ||
| SSIM agreeing. The marginal spread is 70 times the gaudi effect, which is why it | ||
| read as noise. The trade is still worth taking for a median 28% size cut. The | ||
| verdict of "free" was not. |
There was a problem hiding this comment.
For the 0.05 pruning decision, pairing establishes a repeatable change in LPIPS, PSNR, and SSIM, but it does not establish that viewers perceive a quality loss. This document has already explained that movement in these metrics demonstrates sensitivity rather than viewer agreement, so calling the delta a “cost” and deciding it is small enough to trade for file size repeats the unsupported inference the proposed human validation is meant to prevent. Report this as an objective-metric regression pending perceptual validation instead.
Useful? React with 👍 / 👎.
| **PSNR and SSIM really are blind to the Gaussian budget.** This one survives | ||
| pairing and gets stronger for it. Across both cap steps the PSNR interval | ||
| contains zero and the sign splits 3/5, while LPIPS separates every step cleanly. | ||
| Pairing removed the excuse that the effect was hiding in the noise: it is not | ||
| there. |
There was a problem hiding this comment.
The fresh paired analysis only says that the PSNR confidence interval contains zero and that three of five deltas share a sign; failure to reject zero does not show that the metric has no meaningful response. Without reporting the interval and demonstrating that its entire range lies inside a predeclared equivalence bound, potentially relevant positive or negative effects remain compatible with these five frames, so the “really are blind” conclusion is unsupported.
Useful? React with 👍 / 👎.
| They give one number per set, so they cannot be paired the way everything in the | ||
| table above was. Uncertainty has to come from bootstrapping over frames, and the | ||
| paired comparison that made the pruning effect visible has no analogue. |
There was a problem hiding this comment.
When configurations A and B render the same orbit poses and are compared against the same capture set, their KID/CMMD estimates are paired even though each estimate is set-level: bootstrap replicates can resample the same A/B frame indices and the same shared-reference indices before recomputing the metric difference. Claiming that pairing has no analogue encourages independent uncertainty estimates and discards covariance that may be essential for detecting the small pruning effect this section discusses.
Useful? React with 👍 / 👎.
One cleaned 250k export is 55.6 MB as PLY, 5.9 MB as SPZ, 2.6 MB as SOG. Earlier figures in this project came from a cropped export and understated all three.
Three problems, all the document's own argument turned back on it. Calling the pruning regression a "cost" and judging it worth a 28% size cut asserts a perceptual claim from LPIPS movement, which is the exact inference this document says the correlations forbid. It is a repeatable objective regression, and whether it is worth the size cut is what the paired-comparison experiment is for. "PSNR and SSIM are blind" came from five frames failing to reject zero, which is not equivalence. Standardised against each metric's own paired noise the step moves LPIPS 6.1 SD, SSIM 0.77 and PSNR 0.001, so PSNR and SSIM do not even belong together: PSNR barely responds, SSIM responds too weakly to resolve at n=5. Set-level metrics can be bootstrapped as paired after all. Two runs at the same orbit poses against the same capture set share both index sets, so a replicate resamples once and recomputes the difference. Independent bootstraps discard the covariance that made a 0.0007 effect visible.
Ran all three against the capture photographs on two scenes rather than leaving the question at the literature. Results replace the earlier speculation about which one to reach for. Findings that changed the document: - CMMD resolves the 0.05 pruning step in both scenes and agrees with the paired LPIPS result. FID does too. KID resolves only the scene where the effect is eight times larger, which inverts the prediction that its unbiased estimator would make it the safe choice at small n. The binding constraint is variance, not bias. - None of the three orders the whole ladder correctly. FID and KID both produce a resolved step in the wrong direction; CMMD's inversion stays inside an interval spanning zero. - The standard KID protocol averages over independently drawn random subsets, which scored two byte-identical render sets at 15.76 and 16.07. That noise is unpaired and larger than every fine step. The estimator is unbiased on the full set, so the subsetting is dropped. - The 0.00392 rung is a no-op: splatfacto culls at alpha 0.005 during training. Byte-identical renders make it a free check that the bootstrap is index-paired. - Corrected the pruning thresholds in the new sections, which were stated ten times too small. The run names decode as 0.01, 0.02, 0.05 and 0.1, confirmed from the surviving opacity floor in each checkpoint. - Corrected the claim that CMMD is unbiased. The reference implementation clip-mmd vendors is the minimum-variance biased estimator of Gretton et al. Eq. (5).
|
Ran the FID/KID/CMMD experiment the previous revision only speculated about, so the second half of "Would FID or KID do better?" is now measurement rather than prediction. New claims worth checking:
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a684e86932
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Pruning at 0.05 against no pruning is the one comparison here with an | ||
| independent answer, so it is the one to judge them on: |
There was a problem hiding this comment.
The new summary treats the paired LPIPS result as an “independent answer” and uses agreement with it—and presumed monotonic degradation across the finer pruning rungs—to rank FID, KID, and CMMD. Held-out frames make LPIPS a separate measurement, not a ground-truth quality label; this document itself notes that LPIPS movement does not establish what viewers perceive, and no human or otherwise known ordering is provided for the fine opacity ladder. The metrics could therefore be sharing LPIPS's error, so the conclusions that particular steps resolve “backwards” and that CMMD should rank first are unsupported without perceptual labels or a ladder with established ordering.
Useful? React with 👍 / 👎.
The takeaway used agreement with the paired LPIPS result to rank FID, KID and CMMD, and called steps that went the other way "backwards". Neither holds up. LPIPS is a second fallible measurement, not a label, and this document opens by showing it correlates about 0.5 with viewers. No perceptual ordering exists for the fine pruning rungs, so a decrease there is a disagreement between estimators rather than an error by one of them. What is left is a power comparison, which needs no labels: which estimator resolves which step at roughly 100 frames. The one fact about quality in the whole section is the rung that changes nothing, where the true difference is zero by construction. Also records why the question was aimed wrong. Set-level metrics pay off when nothing corresponds to a reference, and scene reconstruction always has a capture. For transforms applied after training the pre-transform export is the better reference: exact, paired per frame, free of transients, and renderable along the delivery path. It cannot decide whether a transform improves anything, since the reference wins by construction, but it does give an ordering that is true rather than assumed.
The distribution-metric renders are JPEG quality 95. Shared across configurations and far below the coarse effects, but at the fine rungs the differences approach the encoder's own error, so some of what those metrics failed to resolve could be the encoding. Says so, and says to write PNG for any rerun aimed at that end. Also marks the "set metrics need no correspondence, which matches the delivery question" claim as the section's premise rather than its finding, since the section goes on to withdraw it.
Tests the claim the previous commit ended on. Each rung scored against the unpruned render at the same poses, per frame. The ladder comes out monotone in both scenes, no neighbouring intervals overlap, and all five rungs separate, where FID, KID and CMMD each inverted somewhere and left the fine end unresolved. Not because LPIPS is the better metric, but because an exact paired reference removes the estimation from a sample of 112 images that was breaking the others. Gives the shipping threshold an actual basis: 0.05 cuts roughly 38% of the Gaussians on both scenes for 0.0055 and 0.0102 LPIPS at 42 dB, while 0.1 cuts 55% and 68% for 0.023 and 0.057. Also corrects why the 0.00392 rung is a no-op. It is not gaudi's training cull at 0.005, which does not apply to r2d2 at all: 0.00392 is 1/255, the cutoff gsplat's rasterizer already applies per pixel, so a Gaussian below it is skipped everywhere and pruning it provably cannot change a render.
The previous numbers came from the q95 JPEGs the render script writes by default, and at the fine end they measured the encoder rather than the pruning. Saving the same checkpoint both ways puts the encoder's own error at 0.00795 LPIPS and 48.5 dB. That is negligible against a photograph, where renders sit around 22 dB, and overwhelming against another render at 61 to 68 dB. The artifacts largely cancel between two nearly identical images and decorrelate as the images separate, so the contamination grows with the rung rather than acting as a constant floor: 17x at 0.01, 6x at 0.02, 1.7x at 0.05. The 0.01 rung read 0.00085 and now reads 0.00005. Corrected shipping figures: 0.05 costs 0.0032 and 0.0078 LPIPS for a 38% Gaussian cut, 0.1 costs 0.020 and 0.054. And the fine rungs are better described than "small": at 61 to 68 dB the renders are not different enough to show anyone.
The first pass scored q95 JPEGs whose encoder error, at 0.00795 LPIPS and 48.5 dB, was larger than the true difference between renders at every rung below 0.05. Redone on lossless renders, with the estimator tables, the sample-size sweep and the conclusions all replaced. What changed. FID's fine-end false positive is gone: it confidently ranked gaudi's 0.01 rung better than unpruned on JPEG, and on faithful renders that step spans zero. That was the encoder, and the earlier claim built on it was wrong. KID still resolves two steps at rungs where the renders differ by 53 to 68 dB, both saying pruning moved the image closer to the photographs. The section no longer ranks the metrics by agreement with LPIPS, which was the standing review finding. It uses a fact instead: the fine rungs render 53 to 68 dB apart, so a resolved interval there is a fault rather than sensitivity. On r2d2's 0.01 to 0.02 step KID and CMMD both resolve with opposite signs, which settles that at least one is wrong without appealing to any perceptual label. Drops the JPEG caveat, now superseded, and corrects the FID sample-size drift to 12.8 points and the 0.05 deviation to 0.0032.
|
Reran everything on lossless renders. The previous numbers were scored on the q95 JPEGs the render script writes by default, and the encoder error (0.00795 LPIPS, 48.5 dB, measured by saving one checkpoint both ways) was larger than the true difference between renders at every rung below 0.05. This addresses the standing P2 about using LPIPS as ground truth. The section no longer ranks the metrics by agreement with LPIPS. It uses a label-free fact: the fine rungs render 53 to 68 dB apart, so a resolved interval there is a fault rather than sensitivity. Worth checking:
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f6d09e0369
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| One pair settles it without any appeal to what a viewer would say. On r2d2's | ||
| 0.01 to 0.02 step KID reports -0.059 with an interval of [-0.104, -0.019] and | ||
| CMMD reports +0.0069 with [+0.0039, +0.0099]. Same images, same frames, both | ||
| intervals excluding zero, opposite signs. At most one of them is right, and | ||
| nothing about the ladder or about human preference is needed to say so. |
There was a problem hiding this comment.
KID and CMMD measure different distances—Inception features with a polynomial kernel versus CLIP features with an RBF kernel, as described in docs/evaluation.md:234-247—so the same image change can legitimately reduce one distance while increasing the other. Without a perceptual label or another declared target, there is no single correct sign, and these intervals therefore do not establish that either metric is wrong or support preferring CMMD later in the section.
Useful? React with 👍 / 👎.
| No opinion scores exist for this ladder, so none of this ranks the three metrics | ||
| by correctness. What it does have is one fact that needs no labels. The renders | ||
| at the fine rungs differ by 53 to 68 dB, measured directly in the section below, | ||
| and two images that close are not different in any way a metric should be able | ||
| to speak confidently about. That is enough to read a resolved interval there as | ||
| a fault rather than as sensitivity. |
There was a problem hiding this comment.
The fresh evidence replacing LPIPS as the presumed ground truth is only a 53–68 dB PSNR range, which does not establish that the render differences are perceptually absent or practically equivalent: PSNR averages error over all pixels and can hide localized structured changes, while a narrow interval can correctly detect an arbitrarily small nonzero effect. Without a predeclared equivalence bound or perceptual labels, a resolved interval at these rungs demonstrates sensitivity, not a fault, so this criterion still cannot rank FID, KID, and CMMD by trustworthiness.
Useful? React with 👍 / 👎.
| The fine rungs land somewhere more useful than "small". At 0.01 the two renders | ||
| differ by 61 dB on gaudi and 68 dB on r2d2, which is not a small difference so | ||
| much as no difference at all. Whatever those rungs cost, it is not something a | ||
| viewer could be shown. That is a defensible | ||
| basis for setting the shipping threshold at 0.05, and it holds regardless of |
There was a problem hiding this comment.
The new export-referenced evidence shows only that the 0.01 rung is extremely close to the unpruned render, but the proposed shipping threshold is 0.05; the table reports substantially larger changes at 0.05 (42–43 dB and LPIPS 0.0032/0.0078), whose perceptual relevance the document explicitly says remains unknown. The behavior at 0.01 therefore provides no basis for accepting the additional pruning between 0.01 and 0.05, so the default still requires validation at the threshold actually being shipped.
Useful? React with 👍 / 👎.
All three review findings hold, and all three are the same mistake in different places: turning a measurement into a verdict without the bound that would license it, which is what this document exists to warn about. Opposite signs are not a contradiction. KID measures a polynomial-kernel distance between Inception features and CMMD an RBF distance between CLIP features, so a change can shorten one and lengthen the other with nothing being violated. The claim retreats to what holds: at most one can track any single notion of quality, and this data cannot say which, or whether either does. A resolved interval at the fine rungs is not a fault. The renders there do differ, so an estimator separating them detects something real, and a narrow interval around a tiny effect is what sensitivity looks like. Calling it an error needs an equivalence bound declared in advance, exactly as this document demands before anyone calls PSNR blind, and no such bound exists. The fine end therefore ranks nobody. What still separates them is about the estimators rather than the images and needs no labels: FID's level drifts 12.8 points across frame counts, more than twice the ladder's range, and KID returns a negative squared distance at 36 frames. CMMD shows neither. That, and only that, is why it is preferred. And the 0.01 rung licenses nothing about shipping at 0.05, where nearly all the deviation and nearly all the saving are. 0.0032 and 0.0078 LPIPS is a magnitude, not a verdict; what changed is that the number exists at all.
|
All three findings hold, and they are the same mistake in three places: turning a measurement into a verdict without the bound that would license it, which is what this document exists to warn about. Fixed in fd34ecc.
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd34ecc1fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| as a number and cannot be compared across differing frame counts. KID returns | ||
| -6.4 at 36 frames, a negative squared distance, which is legal for an unbiased | ||
| estimator and useless for ordering anything. CMMD shows neither pathology. On | ||
| that basis, and only that basis, CMMD is the one to reach for, with the frame | ||
| count held fixed across everything being compared. |
There was a problem hiding this comment.
A negative KID estimate is an expected consequence of using the unbiased U-statistic: it prevents interpreting that point estimate as an absolute squared distance, but does not make paired differences between equally sized configurations “useless for ordering” (the preceding tables do exactly that). Conversely, the documented CMMD V-statistic is biased and nonnegative by construction, so merely observing that it never becomes negative is not evidence that it is more stable or trustworthy. Compare variance or error under matched sample counts before recommending CMMD on this basis.
Useful? React with 👍 / 👎.
| Pruning at 0.1 leaves 446k and 324k, cuts of 55% and 68%, and moves it by 0.020 | ||
| and 0.054 at around 32 dB. The r2d2 cell at 0.1 is the only one that looks like | ||
| a real cost. |
There was a problem hiding this comment.
Fresh evidence after the earlier cost finding is the newly added export-referenced ladder, which now singles out the r2d2 0.1 cell as the only “real cost.” No perceptual or practical bound distinguishes its 0.054 LPIPS/32 dB result from the other cells, and the following paragraph explicitly acknowledges that these measurements cannot say what a viewer notices. Report the magnitude without this verdict unless the planned study supplies such a bound.
Useful? React with 👍 / 👎.
The case for CMMD rested partly on KID returning -6.4 at 36 frames, which supports nothing. A negative value is what an unbiased U-statistic is entitled to produce; it bars reading that point estimate as an absolute distance and leaves the paired differences the tables are built from untouched. CMMD's estimator is the biased V-statistic and nonnegative by construction, so never going negative is its formula rather than a virtue. Replaced with the like-for-like comparison that was missing. Each metric's own bootstrap gives its sampling deviation on the same frames under the same resampling, so a step divided by that deviation compares estimators across units. On the four steps where the renders genuinely differ, CMMD leads in three and KID trails in all four, with FID taking the largest step on r2d2. The fine steps stay out of that table, since a higher ratio there would only mean separating images that barely differ, which is the question no bound has settled. FID's 12.8-point drift in absolute level stands unchanged and independently. Also drops the claim that r2d2 at 0.1 is the only cell that looks like a real cost. Nothing distinguishes it from the others but its size, and the next paragraph already says these numbers cannot speak for a viewer.
|
Correct, fixed in the head commit. The sentence asserted a direction and then admitted in its own next clause that nothing measures it. The direction follows from a model rather than from the data: if viewers share one ordering and differ by noise around it, averaging cancels the noise and leaves a cleaner target, so agreement with the mean is the higher number. That model is not given. Where viewers hold genuinely different orderings the mean represents no one, and a metric that tracks one subgroup can agree with that subgroup more often than with the average, which is your case and it is a real one. The text now states the assumption instead of the conclusion, says no study reports the subject-level numbers that would decide it, and points out that nothing downstream depends on the answer: two times in three against the panel is not settling a close call whichever way the per-viewer correction runs. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 28e367d99d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| matrix built from the centred feature blocks, and the trace term is the sum of | ||
| their square roots. The identity agrees with torchmetrics to three parts in a | ||
| million, with the residual a constant offset that cancels in a difference, and | ||
| 400 replicates now cost less than 25 did. |
There was a problem hiding this comment.
The newly specified procedure says the set-level tables use 4,000 replicates at line 580, but this FID account says the replacement used 400, and line 700 likewise attributes the reported interval to 400. When reproducing the table, those sample counts can produce different percentile tails and R classifications—as the document itself demonstrates for other 400-versus-4,000 runs—so identify which count generated the published FID intervals and make the procedure and narrative agree.
Useful? React with 👍 / 👎.
The section describing the low-rank Frechet identity still ended at 400 replicates and quoted the intervals that count produced, while the tables it feeds now run 4000. It also used the KID fine step as its example of a classification moving from resolved to unresolved, in a direction the paired-U estimator later reverses. The FID example now quotes the interval the tables actually carry, the KID one points at the block sweep where that step is settled, and the speed claim is measured rather than asserted: the identity runs about 1500 times faster per evaluation at 112 frames, so 4000 replicates cost a tenth of what 25 of the old ones did.
|
Correct, fixed in the head commit. That section describes the history of two fixes and never got updated when the tables went to 4000 replicates, so it quoted 400 as if it were current and gave an interval no table carries. The FID example now reads the number the tables actually report: r2d2's 0.01 to 0.02 resolved on 25 replicates at -0.049 with [-0.098, -0.001] and does not once the count can support the tail, [-0.119, +0.012] at 4000. The KID half of that paragraph was worse than stale. It said KID resolves gaudi's 0.00392 to 0.01 under the i.i.d. bootstrap and not under blocks, which is the opposite of what the paired-U estimator does with that step, and it duplicated a case the block sweep now settles. It points there instead. While reconciling the counts I measured the claim about the identity's cost rather than leaving it as an assertion: 12.4 ms against 19.6 s per evaluation at 112 frames on the same machine, so 4000 replicates cost about a tenth of what 25 of the old ones did. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7744cdbabd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Enough of the procedure to rerun it. Each replicate draws ceil(n/L) block | ||
| starts uniformly from 0 to n-1, takes L consecutive indices from each with | ||
| wraparound past the end of the sequence, concatenates them and truncates back | ||
| to n. One index set per replicate indexes the configuration under test, the one |
There was a problem hiding this comment.
For any capture whose final pose is not adjacent to its first, the wraparound rule repeatedly creates synthetic blocks spanning the end-to-start discontinuity; with L=20 on 112–147 frames, this affects a substantial fraction of replicates and can change the reported intervals and R classifications. Restrict starts to blocks that exist in the recorded trajectory, or first establish that each capture path is genuinely cyclic before using a circular moving-block bootstrap.
Useful? React with 👍 / 👎.
The block bootstrap wraps blocks past the end of the sequence, which is what makes it circular in the sense of Politis and Romano and what gives every frame an equal number of blocks to appear in. It also splices the last pose onto the first, and these captures are not loops: gaudi's first and last camera positions are 28.5 median frame steps apart, 61% of the scene's extent, r2d2's are 9.1 steps and 37%. At block length 20 the splice appears in 64% of replicates. Rerunning both tables with starts restricted so no block spans the end moves widths between 25% narrower and 14% wider and swaps which construction excludes zero on four cells, none of which carry an R under either scheme. Every marked step keeps its mark. On the LPIPS ladder the widths move 14% down to 4% up and the smallest neighbouring gap stays above 3.6 interval widths. The tables keep the circular scheme and the document now says what dropping it would cost, which is nothing that changes a conclusion.
|
Good objection, and the premise checks out. Measured and answered in the head commit. The paths are not loops. From the camera positions: gaudi's first and last are 5.66 apart, which is 28.5 median frame steps and 61% of the scene's extent; r2d2's are 3.13 apart, 9.1 steps and 37%. So the wraparound does splice non-adjacent poses. It is also not rare, as you say: at block length 20, 64% of replicates contain a wrapped join, about one per replicate on both scenes. What it is buying. Wrapping is what makes the scheme circular in the sense of Politis and Romano, and it is there so that every frame sits in exactly L blocks. Stopping the starts at What it changes here. I reran both tables with starts restricted so no block spans the end, 4000 replicates, same seeds:
The LPIPS ladder, which uses the same resampler, is the same story: widths move 14% down to 4% up and the smallest gap between neighbouring rungs stays above 3.6 times the wider of the two intervals either way. So the tables keep the circular scheme, which is the standard one and the one with the equal-weight property, and the document now records the non-cyclic capture geometry, how often the splice occurs, and the fact that removing it moves widths without moving a verdict. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cff0cdbd22
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 1. **Transfer check.** Build a ladder on one of our own scenes where the | ||
| ordering is not in doubt: training truncated at 2k, 5k, 10k and 30k steps, or | ||
| pruning at 25/50/75%, degradations large enough that anyone watching the | ||
| orbits agrees which is worse. Require DOVER to reproduce that order. Passing | ||
| shows the published 0.94 survives contact with our content and capture style; | ||
| it does not yet show the metric is useful on close calls. Failing kills it |
There was a problem hiding this comment.
When this check uses training checkpoints or random-pruning levels, their perceptual ordering is not guaranteed merely by the parameter order: later training can regress or overfit, and pruning can remove visible artifacts. The protocol nevertheless treats the asserted ordering as ground truth and discards DOVER if it disagrees, so a valid metric can fail because the ladder itself was mislabelled. Establish this ordering through blinded viewer judgments or use degradations whose ordering is known by construction before applying the pass/fail check.
Useful? React with 👍 / 👎.
| If it holds, adopting it is cheap. These models take a video, need no reference, | ||
| and we already render orbit videos with `ns-render`. The change is one more step | ||
| after export, not a new benchmark. |
There was a problem hiding this comment.
When this experiment is run from the current export artifacts, there are no orbit videos for these video-quality models to score: the repository-wide render paths in scripts/export.sh produce image sequences (--output-format images), and the Python helper also hardcodes image output. Turning those frames into videos is itself metric-visible—especially for models intended to detect compression artifacts—so an unspecified encode can determine the ranking just as JPEG did earlier in this document. Add a pinned, identical, preferably lossless encoding procedure rather than describing metric scoring as the only remaining step.
Useful? React with 👍 / 👎.
Two holes in the proposed study. The video-quality models score a video and the export path renders frames. Both orbit branches in scripts/export.sh pass --output-format images and render_nerfstudio_orbit.py hardcodes the same, so the claim that we already render orbit videos was wrong and the missing step is an encode. That encode is visible to models trained to notice compression, and this document already has a metric resolving a step that turned out to be the JPEG encoder. The protocol now renders losslessly and pins one ffmpeg invocation across every arm, with the delivery-realistic alternative named as a stimulus parameter rather than a detail. The transfer check treated the parameter order of the ladder as the perceptual order and killed DOVER for disagreeing with it. Longer training can regress and pruning removes floaters as readily as detail, which is the direction one of the fine steps in the tables above moves in, so a valid metric could fail because the ladder was mislabelled. The ordering now comes from the same blinded pairwise viewing that step 2 uses, and only the pairs the viewer called unanimously count against the metric.
|
Both correct, fixed in Specify a controlled video-generation step. Verified: The document has already been bitten by exactly this: a metric resolved a step at 61 dB that turned out to be the JPEG encoder rather than the pruning. The protocol now renders the orbit losslessly, the PNG path the tables use, then encodes every arm of every comparison with one ffmpeg invocation differing only in its input, same resolution, frame rate, pixel format, colour range and GOP length, lossless where the model tolerates it. If a study wants delivery-realistic encoding, that is a stimulus parameter and goes in the protocol with its settings, not an afterthought. Establish the transfer ladder's ordering independently. Also right, and the document contradicts itself on it: three sections earlier a pruning step moves a distance downward, which is pruning removing floaters rather than detail, and the same effect can put a 5k checkpoint ahead of a 10k one. Asserting the parameter order as ground truth means a metric that is right about a mislabelled rung fails the gate. Step 1 now takes its ordering from the same blinded pairwise viewing step 2 uses, before DOVER is asked anything, and scores DOVER only on the pairs the viewer called unanimously. Pairs the viewer split carry no information in either direction. It costs a handful of obvious comparisons, which is cheap enough that "costs nothing" was never the reason to skip it. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2944495853
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| """Render Nerfstudio spiral orbit frames with a portable data-path override.""" | ||
| """Render Nerfstudio spiral orbit frames with a portable data-path override. | ||
|
|
||
| Pass --image-format png when the frames will be compared against other renders |
There was a problem hiding this comment.
When orbit frames are produced through the canonical export pipeline, callers cannot follow this advice: the portable-data branch in scripts/export.sh invokes this helper without --image-format, so its parser selects the existing jpeg default, and the export CLI provides no pass-through for the option. Consequently, the normal pipeline path used for a render-to-render experiment still introduces the JPEG contamination documented here as inflating results by up to 19×; either make PNG the helper/export default or expose and pass the format through the pipeline. A repo-wide search found no other caller supplying this flag.
AGENTS.md reference: AGENTS.md:L20-L22
Useful? React with 👍 / 👎.
The helper's docstring told anyone measuring with these frames to pass --image-format png, and nobody going through the pipeline could: export.sh calls render_nerfstudio_orbit.py and ns-render without the flag, both default to JPEG, and there was no pass-through. So the documented contamination, up to 19x on a pruning ladder, was unavoidable on the canonical path. --orbit-image-format reaches both NeRF and splat orbit renders and defaults to jpeg, which is what orbit-frames exists for: frames for a web viewer, where the file size is the point. sdf-render has no image format option, so the SDF path warns and stays JPEG rather than failing.
|
Correct, and it made the docstring's advice impossible to follow on the canonical path. Fixed in Confirmed the whole chain:
With it: README documents the flag and why measurement wants |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34a5e5ea2c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| --load-config "$exp_path/config.yml" \ | ||
| --output-path "$output_path" \ | ||
| --data "$data_path" \ | ||
| --image-format "$orbit_image_format" \ |
There was a problem hiding this comment.
When orbit_frames/ already contains the default JPEG export, rerunning with --orbit-image-format png does not produce the requested lossless set: without --overwrite, should_run_export_dir silently skips the render because the directory is nonempty, while with --overwrite it renders PNGs without deleting the old JPEGs, leaving a mixed directory. This is especially likely when a delivery orbit is later reused for the documented quantitative workflow and can either leave the analysis on JPEG inputs or double/mix its frame population; clear the prior directory on overwrite or make the cache/output path format-aware.
Useful? React with 👍 / 👎.
| add_export_methods "$val" | ||
| i=$((i+2)) | ||
| ;; | ||
| --orbit-image-format) |
There was a problem hiding this comment.
Adding this pipeline flag without updating webui.py leaves the ordinary Web UI orbit workflow permanently on JPEG: there is no format control or build_pipeline_argv parameter, so users can reach the documented lossless path only by discovering and manually typing the flag into the generic expert-arguments box. Keep the UI's command construction and tests aligned with the new export contract as required for flag changes.
AGENTS.md reference: AGENTS.md:L65-L65
Useful? React with 👍 / 👎.
Two holes in the format flag as it shipped. Asking for PNG when the directory already held the JPEG delivery frames skipped the render silently, because the existing cache check only asks whether the directory is non-empty, so the quantitative workflow would have gone on reading the JPEGs it was trying to avoid. Asking for it with --overwrite rendered PNGs alongside the old JPEGs and left one sequence holding both, which a consumer globbing for either reads as a set that is part one and part the other. The orbit path now checks the format of what is already there: a mismatch skips with a message naming it, and --overwrite deletes the previous frames before rendering. The Web UI also gets the control, since going through it was otherwise a permanent JPEG path with no way to the documented lossless one short of typing the flag into the expert arguments box.
|
Both correct, and the first is the more serious of the two. Fixed in Invalidate cached orbit frames when changing format. The flag I added was only half a feature. The orbit path now looks at what is in the directory rather than whether anything is: a format mismatch skips with a line saying the frames are not the requested format, and Expose the orbit format through the Web UI. Also right, and AGENTS.md says so explicitly. Full suite: 285 passed, 2 skipped. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 27e6d64bd7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if export_orbit_image_format: | ||
| cmd_parts.extend(["--orbit-image-format", export_orbit_image_format]) |
There was a problem hiding this comment.
When a Web UI user selects PNG, this adds --orbit-image-format png, but test_cmd() does not include that flag in allowed_export_args, so build_command() reports an invalid export argument and start_run() stops with a command-validation error before launching. Add the flag to the validator so the newly exposed lossless workflow is usable from the UI.
AGENTS.md reference: AGENTS.md:L65-L65
Useful? React with 👍 / 👎.
The dropdown emitted --orbit-image-format and test_cmd's export whitelist did not carry it, so selecting PNG failed validation before the run started. Adding the control without the matching whitelist entry made the UI path worse than it was, since it now offered an option that could only stop the run.
|
Correct, fixed in the head commit. Adding the dropdown without the whitelist entry made the UI path worse than before, since it offered an option whose only effect was to stop the run at validation.
286 passed, 2 skipped. |
|
@codex review |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
ns-evalgives us PSNR, SSIM and LPIPS, and those three numbers have decidedevery splat tuning question so far. Three subjective studies put all three near
0.5 Spearman correlation with human opinion scores on degraded Gaussian splats.
This doc collects what they measured, names which of our conclusions it weakens,
and proposes the cheapest way to check.
Refs: #34 (the doc records the evidence; the issue tracks the experiment)
Changes
docs/evaluation.md: correlation numbers from 3DGS-VBench, MUGSQA, 3DGS-QAand GS-QA; what each study degraded and why the four disagree; the effect on
our cap_max, cleanup threshold, container and frame-count decisions; a
three-step plan starting with DOVER on orbit renders already on disk
README.md: link the new doc from the documentation listType of Change
Testing
make check(docs-only change, no code touched)Checklist