unused var: Remove few unused fields/structs from av2 encoder by SamuthirikaS · Pull Request #5355 · AOMediaCodec/avm

SamuthirikaS · 2026-09-01T14:26:26Z

Intra txb rd hash is removed in commit a7bbe6a. Fields associated with them are now being removed.
Inter txb rd hash is disabled in commit e56083f and removed in commit 968c494. Fields associated with them are now being removed
Remove TXB_RD_INFO/TXB_RD_RECORD subsystem
MB_RD_INFO.*
ModeCosts.*
LV_MAP_COEFF_COST.* is introduced and used by 6780aa9(CWG-E092 Trellis Coded Quantization). TCQ SW cleanup has deleted the file that access this.
PartitionSearchInfo.variance_low
RD_SEARCH_MACROBLOCK_CONTEXT.*
SimpleMotionData.mv_ref
SimpleMotionDataBufs.b_4x64_0/b_64x4_0/b_4x64_1/b_64x4_1
CODING_CONTEXT
RD_RECT_PART_WIN_INFO
MACROBLOCK.skip_mode
encode_b_args.ctx
ToolCfg.enable_cfl_intra
AV2EncoderConfig.operating_points_count
TWO_PASS.frame_stats_next_idx
tcq_param_t.txb_ctx
RestSearchCtxt.tile_y0
QUANT_PARAM.tx_size
PICK_MODE_CONTEXT.parent/index
RefFrameDistanceInfo.nearest_past_ref/nearest_future_ref
RATE_CONTROL.*
EncodeFrameParams.primary_ref_frame
ExternalFlags.cross_frame_context
TPL_SPEED_FEATURES.disable_gop_length_decision

These functions are only needed by the encoder.

Only assign pc_tree->partitioning = partition when last_part_rdc represents a valid RDO search result. Guard the do_recon block so reconstruction is only attempted on a valid partition. Fixes AOMediaCodec#5092 Co-authored-by: Urvang Joshi <urvang@google.com>

…odec#5109) (1) Add checkout step + common setup. This resolves the issue where the 'Compare hashes' step fails with 'fatal: not a git repository' because the git command was run without checking out the repository. (2) Use `github.event.pull_request.head.sha` to get base commit SHA instead of `github.sha` which is the SHA for the "temporary merge commit". This ensures that the `git log` command to find commits containing `STATS_CHANGED` keyword works correctly. Fixes AOMediaCodec#5106 (cherry picked from commit 93fcbc0)

…5098) Drive the main inter-mode loop in av2_rd_pick_inter_mode_sb from a static ref-frame-centric LUT and revise the motion-mode pruning heuristic. STATS_CHANGED Anchor: commit 165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) Speed 0 (cpu-used=0): VCWG CTC (130 frames, all classes, RA and LD). Speed 4 (cpu-used=4): FG16 CTC (33 frames, class A1 and A2, RA) ``` 1) This change in isolation (loop restructure + motion-mode pruning). Speed 1 (cpu-used=1), RA: +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.18 | -0.03 | -0.20 | 0.14 | 95 | 101 | | A2 | 0.18 | 0.67 | 0.35 | 0.20 | 93 | 100 | | Avg w/o B2 | 0.18 | 0.47 | 0.19 | 0.18 | 94 | 100 | +------------+------+-------+-------+------+------+------+ Speed 0 (cpu-used=0), RA: +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.01 | 0.05 | -0.05 | -0.01 | 100 | 101 | | A2 | 0.01 | 0.02 | 0.06 | 0.01 | 100 | 100 | | A3 | 0.00 | 0.01 | 0.10 | 0.01 | 100 | 100 | | A4 | -0.02 | 0.01 | -0.51 | -0.03 | 99 | 100 | | A5 | -0.01 | -0.50 | 0.05 | -0.03 | 101 | 100 | | B1 | 0.05 | 0.18 | -0.04 | 0.04 | 100 | 101 | | B2 | 0.36 | 0.36 | 0.32 | 0.36 | 101 | 108 | | Avg w/o B2 | 0.01 | 0.01 | -0.03 | 0.00 | 100 | 100 | +------------+-------+-------+-------+-------+------+------+ Speed 0 (cpu-used=0), LD: +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A2 | 0.02 | -0.01 | -0.14 | 0.01 | 100 | 100 | | A3 | -0.03 | 0.01 | 0.63 | -0.01 | 101 | 101 | | A4 | -0.00 | -0.26 | -1.13 | -0.05 | 100 | 100 | | A5 | 0.16 | 0.60 | 0.05 | 0.16 | 102 | 103 | | B1 | 0.02 | -0.07 | -0.03 | 0.01 | 101 | 100 | | B2 | 0.36 | 0.34 | 0.61 | 0.38 | 101 | 102 | | Avg w/o B2 | 0.02 | 0.00 | -0.10 | 0.01 | 101 | 100 | +------------+-------+-------+-------+-------+------+------+ 2) For reference, the two follow-up commits (single-ref to compound pruning, and per-ref RD-based pruning for single-ref NEWMV / WARP_NEWMV) combined, Speed 1 (cpu-used=1), RA: +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.20 | -0.08 | -0.06 | 0.16 | 83 | 102 | | A2 | 0.22 | 0.53 | 0.15 | 0.23 | 84 | 101 | | Avg w/o B2 | 0.22 | 0.35 | 0.09 | 0.21 | 84 | 101 | +------------+------+-------+-------+------+------+------+ 3) Overall, motion-mode pruning stacked on top of (2), Speed 1 (cpu-used=1), RA: +------------+------+------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+------+-------+------+------+------+ | A1 | 0.31 | 0.29 | -0.20 | 0.28 | 80 | 101 | | A2 | 0.32 | 0.90 | 0.58 | 0.35 | 80 | 100 | | Avg w/o B2 | 0.32 | 0.72 | 0.35 | 0.33 | 80 | 100 | +------------+------+------+-------+------+------+------+ 4) Overall, motion-mode pruning, single-ref to compound pruning, and per-ref RD-based pruning for single-ref NEWMV / WARP_NEWMV combined, Speed 4 (cpu-used=4), RA: +------------+------+------+------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+------+------+------+------+------+ | A1 | 0.27 | 0.17 | 0.00 | 0.24 | 82 | 102 | | A2 | 0.42 | 0.63 | 0.15 | 0.42 | 81 | 102 | | Avg w/o B2 | 0.38 | 0.49 | 0.11 | 0.36 | 81 | 102 | +------------+------+------+------+------+------+------+ ```

The intra chroma component will not use FSC mode. Simplify the encoding functions therein.

Revive prune_compound_using_single_ref: refresh the per-ref RD snapshot every iteration and prune compound (refs[0], refs[1]) pairs via the single-ref RD cutoff. Enabled at speed >= 1. STATS_CHANGED Anchor: commit AOMediaCodec@165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) ``` +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.10 | -0.15 | -0.27 | 0.07 | 92 | 101 | | A2 | 0.09 | 0.28 | -0.12 | 0.08 | 93 | 101 | | Avg w/o B2 | 0.09 | 0.15 | -0.17 | 0.08 | 93 | 101 | +------------+------+-------+-------+------+------+------+ ```

…odec#5114) Tighten the loop lower bound to pb_mv_precision + 1 so the found cache precision is strictly greater than the current pb_mv_precision, matching the assert's intent.

…iaCodec#5100) Prune single-ref NEWMV / WARP_NEWMV trials whose prior-mode RD for the current ref frame is far from the best across all refs. Gated by prune_newmv_modes_using_prior_rd, enabled at speed >= 1. STATS_CHANGED Anchor: commit AOMediaCodec@165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) ``` +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.14 | -0.04 | -0.14 | 0.11 | 88 | 101 | | A2 | 0.17 | 0.66 | 0.10 | 0.18 | 88 | 101 | | Avg w/o B2 | 0.16 | 0.45 | 0.03 | 0.16 | 88 | 101 | +------------+------+-------+-------+------+------+------+ ```

Adds three single-ref NEWMV DRL search speed features, enabled at speed >= 1: * mv_sf.skip_second_best_subpel: the subpel search normally refines both the best and the second-best full-pel candidate and keeps the lower-variance result; with this feature set, only the best full-pel candidate is refined. * mv_sf.predict_repeated_newmv: when a previously searched ref_mv_idx has a reference MV within one full pel of the current one, reuse the cached NEWMV result and only recompute the MV rate, avoiding the full-pel and subpel search entirely. * mv_sf.newmv_drl_search_limit: once ref_mv_idx reaches the limit, reuse the nearest already-searched result regardless of ref-MV distance instead of running a fresh search. Enabled with a cap of 2. Shares the same search-loop pass as the predict_repeated_newmv reuse. STATS_CHANGED Anchor: commit 165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) Speed 4 (cpu-used=4): FG16 CTC (33 frames, class A1 and A2, RA) ``` 1) Speed 1 a) Overall performance (all three features combined) +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.04 | -0.08 | -0.36 | 0.00 | 99 | 101 | | A2 | 0.04 | -0.04 | -0.18 | 0.03 | 98 | 100 | | Avg w/o B2 | 0.04 | -0.05 | -0.23 | 0.02 | 98 | 100 | +------------+------+-------+-------+------+------+------+ b) Performance of skip_second_best_subpel only +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.10 | -0.32 | 0.09 | -0.11 | 99.5 | 101 | | A2 | -0.01 | -0.09 | -0.34 | -0.02 | 99.5 | 100 | | Avg w/o B2 | -0.04 | -0.16 | -0.21 | -0.05 | 99.5 | 100 | +------------+-------+-------+-------+-------+------+------+ c) Performance of predict_repeated_newmv only +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | 0.00 | -0.08 | 0.08 | 0.00 | 100 | 100 | | A2 | -0.03 | -0.04 | -0.04 | -0.03 | 100 | 100 | | Avg w/o B2 | -0.02 | -0.05 | -0.00 | -0.02 | 100 | 100 | +------------+-------+-------+-------+-------+------+------+ d) Performance of newmv_drl_search_limit only +------------+------+-------+------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+------+------+------+------+ | A1 | 0.05 | -0.17 | 0.09 | 0.03 | 99 | 101 | | A2 | 0.00 | 0.36 | 0.01 | 0.02 | 99 | 100 | | Avg w/o B2 | 0.01 | 0.20 | 0.03 | 0.02 | 99 | 100 | +------------+------+-------+------+------+------+------+ 2) Speed 4 Overall performance (all three features combined) +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.01 | -0.13 | 0.07 | 0.00 | 96 | 101 | | A2 | 0.05 | -0.13 | -0.25 | 0.03 | 96 | 100 | | Avg w/o B2 | 0.04 | -0.13 | -0.15 | 0.02 | 96 | 100 | +------------+------+-------+-------+------+------+------+ ```

For the MultiLayerTest2Embedded2TemporaSframe test, which has dynamic layer dropping, the flag enable_explicit_ref_frame_map_ should be set to true, to avoid DPB desync. Failures were seen at the lower speed settings (e.g., speed 2), not at the speed 5 setting used in the test.

Enables three ref-index based pruning speed features at speed >= 1: - prune_refinemv_by_ref_idx: skip refinemv_loop == 1 for ref-pairs other than (0, 1). - prune_interintra_by_ref_idx: skip INTERINTRA motion mode when ref_frame[0] > 1. - prune_warp_delta_by_ref_idx: skip WARP_DELTA motion mode when ref_frame[0] > 2. STATS_CHANGED Anchor: commit 165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) Speed 4 (cpu-used=4): FG16 CTC (33 frames, class A1 and A2, RA) ``` 1) Speed 1 a) Overall performance (all three features combined) +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.02 | -0.15 | 0.05 | 0.01 | 93 | 101 | | A2 | 0.12 | 0.36 | -0.03 | 0.12 | 93 | 100 | | Avg w/o B2 | 0.09 | 0.21 | -0.00 | 0.09 | 93 | 100 | +------------+------+-------+-------+------+------+------+ b) Performance of prune_refinemv_by_ref_idx only +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.01 | -0.14 | -0.34 | -0.04 | 99 | 101 | | A2 | 0.04 | 0.06 | -0.25 | 0.02 | 99 | 100 | | Avg w/o B2 | 0.02 | 0.00 | -0.28 | 0.00 | 99 | 100 | +------------+-------+-------+-------+-------+------+------+ c) Performance of prune_interintra_by_ref_idx only +------------+------+-------+------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+------+------+------+------+ | A1 | 0.04 | -0.02 | 0.04 | 0.04 | 98 | 100 | | A2 | 0.02 | 0.27 | 0.08 | 0.03 | 98 | 100 | | Avg w/o B2 | 0.03 | 0.18 | 0.07 | 0.03 | 98 | 100 | +------------+------+-------+------+------+------+------+ d) Performance of prune_warp_delta_by_ref_idx only +------------+------+------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+------+-------+------+------+------+ | A1 | 0.03 | 0.00 | 0.12 | 0.03 | 95 | 101 | | A2 | 0.07 | 0.11 | -0.07 | 0.07 | 96 | 100 | | Avg w/o B2 | 0.06 | 0.08 | -0.01 | 0.06 | 96 | 100 | +------------+------+------+-------+------+------+------+ 2) Speed 4 Overall performance (all three features combined) +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.01 | -0.09 | -0.01 | -0.02 | 90 | 101 | | A2 | 0.14 | 0.33 | 0.18 | 0.14 | 89 | 101 | | Avg w/o B2 | 0.09 | 0.20 | 0.12 | 0.09 | 89 | 101 | +------------+------+-------+-------+-------+------+------+ ```

…5103) Extend the intra-luma IST/STX pre-skip and post-trellis gates to cover eob == 0 in addition to the existing eob == 1 (DC-only) case. The eob == 0 branch kills candidates only when stx > 0, so a non-DCT primary with stx == 0 that quantizes to all-zero can still win via skip coding. Add a pre-quant L-infinity gate that predicts primary FP-quant eob = 0 directly from the post-primary coefficients (max |coeff| below the FP quantizer kill threshold min(dq0, dq1) >> (4 + log_scale)). When fired, av2_quant + av2_optimize_b + cost_coeffs are skipped and the same eob_found / DCT_DCT propagation as the post-quant gate is applied. Scoped to intra Y, stx == 0, non-DC-only, IST-enabled, non-QM blocks. Gated behind a new tx_sf.prune_intra_ist_stx_by_zero_eob flag, off by default and enabled at speed >= 1. Encoding at speed 0 is unchanged from the baseline. STATS_CHANGED Anchor: commit dd84ac9 Speed 1 (cpu-used=1): FG16 CTC (33 frames, RA) ``` +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | 0.06 | -0.20 | -0.34 | 0.02 | 98 | 100 | | A2 | 0.09 | -0.18 | -0.12 | 0.07 | 97 | 100 | | A3 | 0.05 | 0.58 | 0.48 | 0.07 | 96 | 99 | | A4 | 0.09 | 0.22 | -3.02 | -0.01 | 97 | 99 | | A5 | -0.12 | -2.35 | 2.22 | -0.12 | 97 | 101 | | B1 | 0.14 | -0.94 | -0.28 | 0.09 | 97 | 101 | | B2 | -0.48 | -0.71 | 0.30 | -0.46 | 97 | 97 | | Avg w/o B2 | 0.07 | -0.32 | -0.24 | 0.04 | 97 | 100 | +------------+-------+-------+-------+-------+------+------+ ```

Make it properly handle cctx cases. Take out coeff dropout to be consistent with what's used in RD search stage. Re-do u plane if its eob is different from that's in RD search to avoid potential mismatch issues due to the change of cctx_type. Tested for A2 and A4 in speed 1 with 33 frames. No coding stats change observed.

The unit tests that exercised base-only implicit-ref-map decode of a multi-layer bitstream (the only path where a TIP-winning base-layer block would fail to reconstruct) were removed in 9f0dfe5 (Fix setting in multi_layers tests, AOMediaCodec#5116), so this encoder-side workaround is no longer needed.

Fixed issue AOMediaCodec#5122

Skip process_compound_inter() function for opfl modes. Tests - cpu-used =1 Anchor: AOMediaCodec@6e71b0c Results (RA): A1 - 17 frames A2 - 33 frames <img width="1066" height="101" alt="MR7" src="https://github.com/user-attachments/assets/7e9e4bce-1065-4d28-84bc-a8df8f3479a7" />

…search (AOMediaCodec#5125) Enable winner transform search Add small fix for winner transform Disable 4 way and 5 way transform in rough search and enable in winner Set winner transform candidates to three Disable existing winner transform speed features. Tests - cpu-used =1 Anchor: 6e71b0c Results (RA): A1 - 17 frames A2 - 33 frames <pre> +---------+-------+--------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+-------+--------+-------+-------+----------+----------+ | A1 | 0.13% | 0.01% | 0.20% | 0.13% | 93.6% | 100% | | A2 | 0.17% | -0.15% | 0.06% | 0.15% | 92.3% | 100% | +---------+-------+--------+-------+-------+----------+----------+ </pre>

Squash merge branch 'leo/inter-sdp-mr' into 'avm-encoder-development' Add a speed level for inter sdp fast method: 1) Early terminate inter-sdp when there is no intra coded blocks in the this region after searching mixed intra and inter region; 2) Early terminate inter-sdp when more than half of the blocks are inter -coded; 3) Early terminate inter-sdp when the best block partitioning is partition-none STATS_CHANGED cpu-use-1 is at commit 6e71b0c is used as anchor. A1 - 17 frames (0.02% loss with 95.3% run-time) A2 - 33 frames (0.09% loss with 94.6% run-time) ``` +---------+-------+--------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+-------+--------+-------+-------+----------+----------+ | A1 | 0.01% | 0.17% | 0.11% | 0.02% | 95.3% | 100% | | A2 | 0.07% | 0.14% | 0.42% | 0.09% | 94.6% | 100% | +---------+-------+--------+-------+-------+----------+----------+ ```

…iaCodec#5137) The `ref_frame_sign_bias` array in the `AV2Common` structure is populated in `av2_setup_frame_sign_bias()` but never used. This patch cleans up the unused array and the associated code.

Enable winner motion mode. Based on tid threshold, choose to perform six parameter warp searches in either winner mode or rough mode. Tests - cpu-used =1 Anchor: AOMediaCodec@6e71b0c Results (RA): A1 - 17 frames A2 - 33 frames ``` +---------+--------+--------+--------+--------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+--------+--------+--------+----------+----------+ | A1 | -0.01% | 0.02% | 0.08% | -0.01% | 98.2% | 100% | | A2 | 0.01% | -0.07% | -0.01% | 0.01% | 98.4% | 101% | +---------+--------+--------+--------+--------+----------+----------+ ```

…nemv (AOMediaCodec#5134) Tested on 962f1c4 at speed 1, RA 33 frames +------------+-------+-------+-------+-------+-------+ | Class | Y | Cb | Cr | wAvg | Enc% | +------------+-------+-------+-------+-------+-------+ | A1 | +0.06 | +0.01 | +0.07 | +0.06 | 97.06 | | A2 | +0.08 | -0.08 | +0.03 | +0.07 | 96.94 | | Avg | +0.07 | -0.05 | +0.04 | +0.06 | 96.98 | +------------+-------+-------+-------+-------+-------+

Reuse model RD when one of the following conditions are met. - Current mode is the same as the cached mode and at least one of the reference frame is the same as the cached one - All the reference frames is the same as the reference frame of the cached mode Otherwise, skip processing current mode or reference frame STATS_CHANGED Results are tested on top of 6e71b0c with cpu-used = 1. ``` +---------+-------+--------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+-------+--------+-------+-------+----------+----------+ | A1 | 0.03% | -0.05% | 0.00% | 0.03% | 97.8% | 100% | | A2 | 0.03% | 0.41% | 0.13% | 0.05% | 96.6% | 100% | +---------+-------+--------+-------+-------+----------+----------+ ```

This pull request refactors search_tx_type() to improve code readability, reduce search loop complexity, fix compilation errors under debug configuration (COLLECT_TX_SIZE_DATA).

Increase TCQ zero threshold to 9 * abs_coef >= 10 * dequant when base_qindex < 185. Anchor: AOMediaCodec@6e71b0c Tests: cpu-used =1, RA A1 - 17 frames A2 - 33 frames ``` +---------+--------+--------+--------+--------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+--------+--------+--------+----------+----------+ | A1 | -0.01% | -0.01% | 0.00% | -0.01% | 98.6% | 98.6% | | A2 | 0.00% | -0.06% | -0.05% | 0.00% | 98.9% | 100.1% | +---------+--------+--------+--------+--------+----------+----------+ ```

As extended sdp is allowed only for partition type less than uneven 4 way search partition, this change has no effect on stats. This is done for consistency.

This change unifies uneven 4way partition search functions and h partition search functions into one function No stats changed.

- Search across ext partitions is rolled into a loop No stats changed

- Moved pruning using ml results logic to a function - Removed duplicate/unused initializations - Handled a case where none partition search gets skipped if search_none_after_rect is false and search_none_after_split is true and search none is delayed to after rect based on split results - Moved terminate partition search after none search into none search function so that it gets called correctly after none evaluation - Restored rdmult original in an early return case

Pulled common calculations from the pixel filtering loop and conducted them beforehand. CTC tests showed no stats change.

)

Added a speed feature to early terminate the warp delta refinement loop if the rate-distortion cost improvement in a pass drops below 5%. Results for RA CTC, A1 17 frames, A2 33 frames, speed 1: (Anchor: 7c9dfba) ``` +-----+-------+-------+-------+-------+---------------+ |Class| Y | Cb | Cr | wAvg |EncInstCount(%)| +-----+-------+-------+-------+-------+---------------+ | A2 |-0.0004|-0.0367|-0.2123|-0.0103| 98.93 | | A1 |-0.0649| 0.1274| 0.0844|-0.0513| 98.89 | +-----+-------+-------+-------+-------+---------------+ ``` STATS_CHANGED for speed >= 1

Enable frame level bawp based on histogram. Enabled for speed 2 and above. 33 frame RA results for speed 2 and 3 are as follows: +---------+--------+-------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+-------+-------+-------+----------+----------+ Speed2, Overall w/o B2 (RA) +0.00% +0.01% +0.01% +0.00% 99.51% 100.12% Speed3, Overall w/o B2 (RA) +0.00% +0.01% +0.03% +0.00% 99.45% 99.53% +---------+--------+-------+-------+-------+----------+----------+

…c#5314) The expensive pixel-domain distortion path in search_tx_type() is skipped if the lightweight transform-domain RD estimate already exceeds best_rd or ref_best_rd. This is controlled by a new speed feature flag skip_pixel_dist_calc_using_tx_dist, enabled for speed >= 1. STATS_CHANGED for speed >= 1 Test results (RA) for Speed 1 Anchor: commit 5d628d8 A1 - 17 frames A2 - 33 frames ``` +------------+-------+-------+-------+-------+-------------+ | Class | Y | Cb | Cr | YUV |EncInstCount%| +------------+-------+-------+-------+-------+-------------+ | A1 | -0.03 | 0.43 | -0.06 | 0.00 | 95.84 | | A2 | 0.04 | 0.46 | -0.15 | 0.05 | 97.35 | +------------+-------+-------+-------+-------+-------------+ ```

Adds the build-system, RTCD, and runtime-dispatch plumbing to host AVX-512 kernels in libavm. Infrastructure only — no kernels are added and nothing is `specialize`d to `avx512`, so this is a runtime no-op till follow-up kernel PRs land. Includes: runtime detection in `x86.h` (XGETBV opmask/ZMM state + CPUID F/DQ/BW/VL), RTCD `avx512` arch (x86_64 only), CMake object-library hooks and MSVC `/arch:AVX512` mapping, `HAVE_AVX512`/`ENABLE_AVX512` (default ON, inherits the disable cascade), and a gtest CPU gate. Reversible via `-DENABLE_AVX512=OFF`. --------- Signed-off-by: Nithya V S <Nithya.VS@amd.com>

Precompute decay, normalization, motion-distance, and block-error factors outside the per-pixel loops. Align SSE2 implementation with C function to support 64x64 blocks (TF_BLOCK_SIZE = BLOCK_64X64, 16 subblocks), enabling SSE2 SIMD dispatch in the temporal filter. Instantiate HBDTemporalFilterTest for C vs SSE2 in test/temporal_filter_test.cc. Unit tests show that SSE2 is ~2.6x faster than C (61-62% reduction), 1.5-2x faster than the original SSE2. [ RUN ] SSE2/HBDTemporalFilterTest.DISABLED_Speed/0 ref_time=2562927 us | tst_time=1012646 us | gain=2.53x (60.5% reduction) width=64 height=64 ref_time=2317598 us | tst_time=881671 us | gain=2.63x (62.0% reduction) width=64 height=64 ref_time=2431593 us | tst_time=954904 us | gain=2.55x (60.7% reduction) width=64 height=64

Replace per-frame calloc/free in av2_ccso_search with persistent buffers on AV2_COMP, freed at encoder close. No bitstream impact.

Bias towards large partition block is enabled for high QP area in low complexity decode (LC) mode. Only enabled for >=2160p now, CTC performance (65 frames, speed 1): baseline commit: 5d628d8 ``` Test_Set PSNR-YUV SSIM VMAF ENC_TIME DEC_TIME A1 0.04% -0.24% -0.02% 97% 96% ``` STATS_CHANGED only for LC mode

A speed feature `include_dip_for_top_n_model_rd_pruning` was recently added to speed up DIP for for speed >= 1. Because of this, there is a very small slowdown when using DIP, with a good coding gain at speed >= 3 (where DIP was disabled before). Anchor commit: 44be072 (Aug 25, 2026) ### Speed 4 CTC 33 frames RA: ``` +---------+--------+-------------+ | Summary | YUV | Enc-time | +---------+--------+-------------+ | A1 | -0.85% | 101.79% | | A2 | -0.53% | 102.00% | |Avg wo B2| -0.46% | 102.56% | +---------+--------+-------------+ ``` ### Speed 3 CTC 33 frames RA: ``` +---------+--------+-------------+ | Summary | YUV | Enc-time | +---------+--------+-------------+ | A1 | -0.76% | 102.47% | | A2 | -0.46% | 103.18% | |Avg wo B2| -0.43% | 103.75% | +---------+--------+-------------+ ``` STATS_CHANGED for speed >= 3

) - Replace the RESTORATION_LINEBUFFER_WIDTH based static buffers, sized for a max supported picture with buffers allocated per plane from the actual frame width. - Replace per RU alloc and free in favor of per frame allocation no stats changed

Added a speed feature to terminate warp MV refinement iterations when the improvement over the previous iteration is below 5%. Enabled for speed >= 3. Results for RA CTC, A1 17 frames, A2 33 frames, speed 3: (Anchor: 735d104) ``` +-----+------+-------+-----+-------+---------------+ |Class| Y | U | V | YUV |EncInstCount(%)| +-----+------+-------+------+------+---------------+ | A1 | 0.04 | -0.28 | 0.05 | 0.02 | 97.9 | | A2 | 0.04 | -0.21 | 0.10 | 0.02 | 98.0 | +-----+------+-------+------+------+---------------+ ``` STATS_CHANGED for speed >= 3

This change unifies the rectangular sub-block search to improve the readabilty and avoid code duplication. Unified code is extracted to new function has_searched_rect_subblock() which scans HORZ/VERT sub-blocks for a previously searched partition. No stats changed Change-Id: I331cf4a82fadc549122a91f2acb7f1e9eef1b2cd

- Abstracted the TPL stage inter/intra cost accumulation to calculate_cost_from_tpl_data(). - Abstracted the IntraBC prediction code to try_intrabc_after_inter_search(). No stats changed Change-Id: Icc70b559709078236cc96b4d3af34c5d10cff6ac

- Hoist the frame/sequence level reads out of the mode loop into locals. - Fold the loop-invariant BLOCK_4X4 check into the mode loop bound. - Flatten the BRU gate from three nesting levels to two. No stats changed Change-Id: I7d9e8fa4794e1959185e9aa39aedd7aad0cfc319

- Drop is_comp_mode, a second name for comp_pred used interchangeably with it in the same loop body. - Drop the no-op guard around the search_state.best_rd assignment. - Compute num_amvd_modes and amvd_inverted just above the use_amvd loop that consumes them. - Scope the loop counter i to the loops that use it. - Initialize the motion mode prune pool with a single flat loop. - Use the local aliases consistently (e.g: sf-> over cpi->sf etc). - Add the missing const to the pointer locals that are never reassigned. - Move the main loop and hdres threshold comments next to the code they describe. - Use prefix increment in the two loops that used postfix. No stats changed Change-Id: Ic88588aa0ce5986bdabb471cf4322141975a00c5

Gcc reports "array subscript 8 is above array bounds of unsigned int[8][8]" for ref_costs_comp[ref_frame_index]. The index only reaches TIP_FRAME_INDEX(8) for TIP, which is single reference only, so the compound branch is never evaluated with that index. ref_costs_single is sized to include the TIP slot, ref_costs_comp is not. Added the bounds check to the condition so that the invariant is visible to the compiler. No stats changed Change-Id: I518f7414c5958f886e6750ee7402a48ecc1a895a

) This newly activates fast two pass partition for speed=2, while improving speed-quality ratio for speed=3/4. At speed=2 it is turned on only for class A1 (2160p and above). Class A2 (2K) lost coding efficiency at speed=2, so we leave it off there. Speeds 3 and 4 are not changed and stay on for every resolution. Some of the partition ML/pruning schemes interact poorly with the fast two-pass partition search, so skip them in the dry pass and on trusted wet-pass blocks. The full policy across regimes: * FAST dry pass: disabled. Shape is picked from reduced-tool RD, and the dry pass is already fast. * FAST wet pass, forced: disabled. Partition is already fixed by the dry-pass template; the decision is discarded. * FAST wet pass, reopened: enabled. Block is genuinely searched. * CONSERVATIVE / one-pass: enabled. Predicate is inert on this path. Anchor: commit 7361d54 Speed 4 (cpu-used=4): FG16 CTC (33 frames, class A1 and A2, RA) Speed 3 (cpu-used=3): FG16 CTC (33 frames, class A1 and A2, RA) Speed 2 (cpu-used=2): FG16 CTC (33 frames, class A1 and A2, RA) ``` 1) Speed 4 (cpu-used=4) +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.73 | 0.26 | -0.05 | -0.66 | 106 | 101 | | A2 | -0.49 | 0.22 | 0.20 | -0.43 | 105 | 101 | | Avg w/o B2 | -0.56 | 0.23 | 0.12 | -0.50 | 105 | 101 | +------------+-------+-------+-------+-------+------+------+ 2) Speed 3 (cpu-used=3) +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.75 | 0.08 | 0.17 | -0.68 | 106 | 101 | | A2 | -0.47 | 0.18 | -0.15 | -0.43 | 104 | 100 | | Avg w/o B2 | -0.55 | 0.15 | -0.05 | -0.51 | 105 | 100 | +------------+-------+-------+-------+-------+------+------+ 3) Speed 2 (cpu-used=2) +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.50 | -2.10 | -2.72 | -0.65 | 84 | 87 | | A2 | 0.46 | 1.97 | 1.60 | 0.56 | 83 | 95 | | Avg w/o B2 | 0.17 | 0.76 | 0.32 | 0.20 | 84 | 93 | +------------+-------+-------+-------+-------+------+------+ ``` STATS_CHANGED

clean up the setting of prune_ref_mv_idx_search. NO_STATES_CHANGE

) Add a unified SMS pre-screener (sms_unified_prune) that runs one MLP inference per square block before the partition search and uses the output to prune partition candidates. Anchor: da53762 Test condition: CTC, RA, 33 frames, speed 4 RA: ``` +----------------+------+------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | +----------------+------+------+------+------+------+ | A1 | 0.01 | 0.35 |-0.09 | 0.02 | 96.3 | | A2 | 0.14 | 0.14 | 0.27 | 0.14 | 95.5 | | Average w/o B2 | 0.09 | 0.12 | 0.29 | 0.10 | 96.7 | +----------------+------+------+------+------+------+ ``` STATS_CHANGED

Previously, TCQ evaluated coefficients along anti-diagonal scan lines in a portable C loop (trellis_loop_diagonal_st8), invoking individual fine-grained RTCD SIMD functions per coefficient. This fine-grained dispatch structure incurred significant overhead: - Repeated RTCD function calls and indirect branches on every coeff. - Register spills to memory structures across subroutine boundaries. - Optimization barriers preventing compiler vector scheduling across the full diagonal loop. This patch introduces a coarse-grained, block-level RTCD entry point: av2_trellis_loop_diagonal_st8() with AVX2 specialization (av2_trellis_loop_diagonal_st8_avx2). Updates are now executed contiguously within AVX2 registers without exiting to scalar C code per coefficient. Unit tests are added to compare the the new and original implementation. We can see that the TCQ function becomes 3-10% faster. ``` ======================================================================= TX Size Pure C (us) Base AVX2 (us) Patch AVX2 (us) vs Base AVX2 ======================================================================= 4x4 110313 45646 44360 1.03x (+2.8%) 8x8 369747 102636 96537 1.06x (+5.9%) 16x16 343618 81354 74201 1.10x (+8.8%) 32x32 268794 60375 54646 1.10x (+9.5%) 4x8 195707 64939 61781 1.05x (+4.9%) 8x4 201683 65227 61872 1.05x (+5.1%) 8x16 175484 44150 40669 1.09x (+7.9%) 16x8 180465 44455 40943 1.09x (+7.9%) 16x32 134924 30975 28011 1.11x (+9.6%) 32x16 134311 30946 28192 1.10x (+8.9%) 4x16 364581 103343 96494 1.07x (+6.6%) 16x4 390933 104326 97702 1.07x (+6.3%) 8x32 343005 81586 74379 1.10x (+8.8%) 32x8 352547 81916 74820 1.09x (+8.7%) ======================================================================= ```

In this PR, when encoder flag reduce_max_drl_refmvs is set to 1 (i.e. speed >= 2), the encoder searches less reference MV candidate for the compound mode with separate DRL. Test results of 33f RA (speed 2), based on commit [ae462ed](AOMediaCodec@ae462ed), are as folllows: PSNR-Y PSNR-U PSNR-V PSNR-YUV EncTime DecTime A1: +0.04% +0.01% +0.21% +0.05% 96.47% 98.30% A2: +0.07% +0.28% -0.05% +0.07% 96.48% 98.14%

- Intra txb rd hash is removed in commit a7bbe6a. fields associated with them are now being removed. - Inter txb rd hash is disabled in commit e56083f and removed in commit 968c494. fields associated with them are now being removed - remove TXB_RD_INFO/TXB_RD_RECORD subsystem - MB_RD_INFO.* - ModeCosts.* - LV_MAP_COEFF_COST.* is introduced and used by 6780aa9(CWG-E092 Trellis Coded Quantization). TCQ SW cleanup has deleted the file that access this. - PartitionSearchInfo.variance_low - RD_SEARCH_MACROBLOCK_CONTEXT.* - SimpleMotionData.mv_ref - SimpleMotionDataBufs.b_4x64_0/b_64x4_0/b_4x64_1/b_64x4_1 - CODING_CONTEXT - RD_RECT_PART_WIN_INFO - MACROBLOCK.skip_mode - encode_b_args.ctx - ToolCfg.enable_cfl_intra - AV2EncoderConfig.operating_points_count - TWO_PASS.frame_stats_next_idx - tcq_param_t.txb_ctx - RestSearchCtxt.tile_y0 - QUANT_PARAM.tx_size - PICK_MODE_CONTEXT.parent/index - RefFrameDistanceInfo.nearest_past_ref/nearest_future_ref - RATE_CONTROL.* - EncodeFrameParams.primary_ref_frame - ExternalFlags.cross_frame_context - TPL_SPEED_FEATURES.disable_gop_length_decision Co-authored-by: Satheesh Kumar <satheesh.kumar@ittiam.com>

SamuthirikaS · 2026-09-02T17:17:38Z

@yunqingwang1 @urvangjoshi
/cc @vikasprasad10 @harishdm @ranjit-tulabandu @cherma-a @deepa-kg @RemyaPrakasan @ram-mohan
We have verified that this PR is bit-exact for speeds 0 to 3 across RA, LD, and AI configurations wrt commit 2ab55a5. Please review.

urvangjoshi · 2026-09-02T23:43:11Z

av2-enc branch itself was updated (rebased). Pls rebase your PR.

chengchen-google and others added 30 commits July 1, 2026 14:07

Move set_sb_size() to encoder dir (AOMediaCodec#5084)

dd84ac9

Move get_partition() to enc dir (AOMediaCodec#5091)

d11d079

These functions are only needed by the encoder.

Remove FSC path from intra joint UV function (AOMediaCodec#5111)

ba3177b

The intra chroma component will not use FSC mode. Simplify the encoding functions therein.

Fix latent assert in handle_newmv MV-precision cache lookup (AOMediaC…

39e6198

…odec#5114) Tighten the loop lower bound to pb_mv_precision + 1 so the found cache precision is strictly greater than the current pb_mv_precision, matching the assert's intent.

Add cctx type array update back (AOMediaCodec#5123)

1d2febc

Fixed issue AOMediaCodec#5122

Clean up unused ref_frame_sign_bias[] in AV2Common structure (AOMed…

b095600

…iaCodec#5137) The `ref_frame_sign_bias` array in the `AV2Common` structure is populated in `av2_setup_frame_sign_bias()` but never used. This patch cleans up the unused array and the associated code.

Refactor search_tx_type() (AOMediaCodec#5136)

6cf7ad4

This pull request refactors search_tx_type() to improve code readability, reduce search loop complexity, fix compilation errors under debug configuration (COLLECT_TX_SIZE_DATA).

Fix typo during checks for allowing extended sdp

82e0f6d

As extended sdp is allowed only for partition type less than uneven 4 way search partition, this change has no effect on stats. This is done for consistency.

Unify extended partition search functions

6a5151d

This change unifies uneven 4way partition search functions and h partition search functions into one function No stats changed.

Refactor ext partition search in av2_rd_pick_partition

3855f5d

- Search across ext partitions is rolled into a loop No stats changed

yunqingwang1 and others added 25 commits August 25, 2026 08:51

Temporal filter function optimization

33e17b1

Pulled common calculations from the pixel filtering loop and conducted them beforehand. CTC tests showed no stats change.

Promote the uneven-4-way partition disable to speed 1 (AOMediaCodec#5305

cc53250

)

Pre-allocate CCSO search buffers (AOMediaCodec#5323)

ae462ed

Replace per-frame calloc/free in av2_ccso_search with persistent buffers on AV2_COMP, freed at encoder close. No bitstream impact.

Refactor rd_pick_intra_dip_sby() (AOMediaCodec#5306)

bfb7f41

Add 1-pass real-time CBR rate control support (AOMediaCodec#5328)

6a36d4e

clean up setting of prune_ref_mv_idx_search (AOMediaCodec#5344)

18a0096

clean up the setting of prune_ref_mv_idx_search. NO_STATES_CHANGE

aom-bot added the cla-signed label Sep 1, 2026

SamuthirikaS marked this pull request as ready for review September 2, 2026 17:16

urvangjoshi force-pushed the av2-enc branch from 2128e7e to cfea2c7 Compare September 2, 2026 22:23

Navigation Menu

Uh oh!

unused var: Remove few unused fields/structs from av2 encoder - #5355

unused var: Remove few unused fields/structs from av2 encoder#5355
SamuthirikaS wants to merge 181 commits into
AOMediaCodec:av2-encfrom
SamuthirikaS:cleanup

SamuthirikaS commented Sep 1, 2026

Uh oh!

SamuthirikaS commented Sep 2, 2026

Uh oh!

urvangjoshi commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Uh oh!

Conversation

SamuthirikaS commented Sep 1, 2026

Uh oh!

SamuthirikaS commented Sep 2, 2026

Uh oh!

urvangjoshi commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants