Tune variance partition thresholds in RTC - #5256
Open
jianj-g wants to merge 1 commit into
Open
Conversation
jianj-g
marked this pull request as draft
August 12, 2026 19:08
jianj-g
force-pushed
the
rtc-vbp-tune-thresh
branch
6 times, most recently
from
August 31, 2026 16:27
5cd7cf7 to
7367c88
Compare
Comprehensive overhaul of variance-based partition (VBP) logic and
thresholds for real-time (RTC) encoding mode across resolutions
(low-res, 360p, VGA, 720p, 1080p+) and content types (camera video
and screen content):
1. Exponential quantizer modeling:
Update threshold_base calculation to properly model AV2's exponential
quantizer step size across QP, replacing the linear AC quantizer scale.
2. Resolution-adaptive thresholds and high-QP scaling:
Refine thresholds for <= 288p, 360p, 720p, and 1080p+, introducing
smooth high-QP threshold slopes to avoid over-partitioning fine textures
and UI elements at coarse quantizers.
3. Sub-block variance spread evaluation & partition gating:
- Evaluate sub-block variance spread in get_part_eval_based_on_sub_blk_var
to avoid unnecessary 8x8 splits on flat/uniform areas while preserving
splits for fine details.
- Extend 32x32 sub-block variance difference checking to non-screen
resolutions > 360p (VGA, 720p) guarded by max sub-block variance,
avoiding trapped texture details in 32x32 blocks.
- Guard 64x64 block splitting based on maximum sub-block variance.
4. Screen content tuning (is_screen):
- Guard 32x32 variance-diff force-split with !is_screen to retain large
32x32 and 16x16 blocks needed for Palette Mode and IntraBC efficiency.
- Tailor sub-block split thresholds and high-QP scaling specifically for
screen content.
Benchmark results vs upstream baseline:
rtc (28 clips, VGA/720p camera video):
vmaf: -5.3% | ovr_psnr: -3.7% | speedup: +4.8%
rtc_derf (20 clips, <= 288p low-res):
vmaf: -3.6% | ovr_psnr: -2.1% | speedup: +1.1%
rtc_screen (9 clips, screen content):
vmaf: -8.4% | ovr_psnr: -3.3% | speedup: +2.2%
STATS_CHANGED for RTC mode
TAG=agy
CONV=fe11822c-44c1-4487-85e3-edf6281da4a9
jianj-g
force-pushed
the
rtc-vbp-tune-thresh
branch
from
August 31, 2026 17:04
7367c88 to
f74f37b
Compare
jianj-g
marked this pull request as ready for review
August 31, 2026 17:36
marpan-max
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Comprehensive overhaul of variance-based partition (VBP) logic and
thresholds for real-time (RTC) encoding mode across resolutions
(low-res, 360p, VGA, 720p, 1080p+) and content types (camera video
and screen content). This only affects RTC mode with speed >= 6.
Exponential quantizer modeling:
Update threshold_base calculation to properly model AV2's exponential
quantizer step size across QP, replacing the linear AC quantizer scale.
Resolution-adaptive thresholds and high-QP scaling:
Refine thresholds for <= 288p, 360p, 720p, and 1080p+, introducing
smooth high-QP threshold slopes to avoid over-partitioning fine textures
and UI elements at coarse quantizers.
Sub-block variance spread evaluation & partition gating:
to avoid unnecessary 8x8 splits on flat/uniform areas while preserving
splits for fine details.
resolutions > 360p (VGA, 720p) guarded by max sub-block variance,
avoiding trapped texture details in 32x32 blocks.
Screen content tuning (is_screen):
32x32 and 16x16 blocks needed for Palette Mode and IntraBC efficiency.
screen content.
Benchmark results vs upstream baseline:
rtc (28 clips, VGA/720p camera video):
vmaf: -5.3% | ovr_psnr: -3.7% | speedup: +4.8%
rtc_derf (20 clips, <= 288p low-res):
vmaf: -3.6% | ovr_psnr: -2.1% | speedup: +1.1%
rtc_screen (9 clips, screen content):
vmaf: -8.4% | ovr_psnr: -3.3% | speedup: +2.2%
STATS_CHANGED for RTC mode