Fix transposed BT.709 to BT.2020 matrix in Vulkan HDR imgui overlay

[emansom]

Problem

The ImGui overlay picks up a colour tint on Vulkan whenever it's drawn onto an HDR swapchain (HDR10 ST2084 / PQ, and the HLG path as well). Neutral greys visibly shift toward green. The Direct3D backends are unaffected.

Cause

res/shaders/imgui_hdr.hlsl is compiled both as HLSL (with fxc) and as GLSL (with glslangValidator), so it carries a small compatibility shim that only activates for the GLSL builds, where float3x3 is #defined to mat3:

#ifdef float3x3
    #define mul(a, b) (a) * (b)
#endif
...
const float3x3 bt709_to_bt2020 = float3x3( /* 9 coefficients, one row per line */ );
return mul(bt709_to_bt2020, col);

The two languages read that same list of coefficients differently. HLSL builds matrices in row-major order [1], and mul(M, v) treats v as a column vector [2], so mul(bt709_to_bt2020, col) produces the intended M * col. GLSL fills matrix constructors in column-major order instead (spec §5.4.2 [3]), so the identical coefficients build the transpose, M^T. The shim's (matrix) * (vector) is then M * col with a right-hand vector treated as a column vector (spec §5.9 [3]), which comes out as M^T * col, i.e. the matrix applied transposed.

A transposed BT.709 to BT.2020 conversion no longer preserves the D65 white point, so a neutral grey (g, g, g) ends up around (0.71, 1.34, 0.95) * g, which reads as green. Both to_hdr10_pq and to_hdr10_hlg call bt709_to_bt2020, so both are affected. The sRGB and scRGB paths never touch the matrix, so they stay correct.

Fix

Swap the operands in the GLSL shim:

#define mul(a, b) (b) * (a)

The multiply is now col * M. In GLSL a left-hand vector is treated as a row vector (spec §5.9 [3]), so col * M evaluates to M^T * col, which cancels the constructor transpose and reproduces HLSL's row-major M * col. The Microsoft docs put it plainly: "swapping the order of the arguments supplied to multiply is equivalent to transposing the matrix" [1]. res/shaders/imgui_ps_450.spv is regenerated with the existing glslangValidator flags.

The change is about as contained as it gets. Diffing the disassembly of the old and regenerated SPIR-V, the only meaningful difference is a single instruction (everything else is just ID renumbering), and the matrix coefficients are byte-for-byte identical:

- %60 = OpMatrixTimesVector %58 %59
+ %60 = OpVectorTimesMatrix %45 %59

So the color_space switch, to_scrgb, the SDR path, the PQ and HLG transfer functions, the texture sampling and the blending are all left untouched.

Scope by backend

• Direct3D 9/10/11/12: #ifdef float3x3 is false here (it's a builtin type, not a macro), so the shim never activates and HLSL's own row-major mul was always correct. imgui_ps_3_0.hlsl and imgui_ps_4_0.hlsl are unchanged, and this backend never had the bug.
• Vulkan (Windows and Linux): the bug lives purely in the shader math, so every Vulkan HDR10 user ran into it, not only Proton. It's fixed through imgui_ps_450.spv.
• OpenGL 4.30: imgui_ps_430.glsl doesn't include imgui_hdr.hlsl, so there's no matrix path there and nothing changes.
• HLG: fixed by the same one-line change. It differs from PQ only in the transfer function (hlg_inverse_eotf) and the reference-white scaling (/1000 rather than /10000), neither of which involves the matrix.

Testing

Tested on Arch Linux with KDE Plasma (Wayland) and HDR enabled, on an HDR10 ST2084 Vulkan swapchain (GTA IV running through D3D9 and DXVK). After the change the overlay's neutral greys come out neutral rather than green. I confirmed this with RenderDoc captures taken before and after the fix, where the overlay pixels move from green-dominant (R, B < G) to neutral (R ≈ G ≈ B). PQ is visually confirmed. HLG goes through the identical code, though I haven't checked it on an HLG display.

Screenshots

Before, the green-tinted overlay on the HDR10 ST2084 Vulkan swapchain:

After, the neutral overlay once the fix is applied:

References

1. HLSL, Per-Component Math Operations ("Matrix Ordering"): constructors "always follows row-major ordering", and "swapping the order of the arguments supplied to multiply is equivalent to transposing the matrix." https://learn.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-per-component-math
2. HLSL, mul: "If y is a vector, it [is] treated as a column vector." https://learn.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-mul
3. OpenGL Shading Language 4.60 spec: §5.4.2 (matrix constructors "constructed and consumed in column major order") and §5.9 (a right vector operand "is treated as a column vector and a left vector operand as a row vector"). https://registry.khronos.org/OpenGL/specs/gl/GLSLangSpec.4.60.html