Fix Windows export hanging in finalize when audio and video end apart

[ruccho]

Summary

On Windows, StopAndExportAsync can wait forever: the exported mp4 is left as ftyp / mdat with no moov, and the session never completes. It happens when the audio and video in the exported segment end more than about 1.3–1.4 s apart, so it shows up intermittently in real projects.

The cause is that PlaceMarker(MFSTREAMSINK_MARKER_ENDOFSEGMENT, NULL, NULL) is rejected with MF_E_INVALIDTYPE on recent Windows builds (the failure reported in #156). Since #157 the rejection is treated as non-fatal, but the MPEG-4 sink then never learns that the stream has ended, and with a large enough end gap it stops requesting samples from the other stream. That stream never drains, finalization never starts, and the export hangs.

The rejection only happens when pvarContextValue is NULL or VT_EMPTY. With a non-empty context value the marker is accepted and MEStreamSinkMarker follows. This PR passes one, and waits for MEStreamSinkMarker before finalizing.

Root cause

1. unienc_windows_mf places the end-of-segment marker as PlaceMarker(ENDOFSEGMENT, NULL, NULL).
2. On Windows 11 24H2 / 25H2 (mfmp4srcsnk.dll 10.0.26100.8457 and later; verified on 10.0.26100.8875), the MPEG-4 stream sink returns MF_E_INVALIDTYPE whenever pvarContextValue is NULL or VT_EMPTY. The documentation allows NULL, and this HRESULT is not among the documented return values. Every marker type and every pvarMarkerValue variant we tried fails the same way, which matches Windows: export always fails with "Failed to complete muxer: channel closed" — OS MPEG4 sink rejects PlaceMarker (MF_E_INVALIDTYPE) #156.
3. The failure is logged as non-fatal and the stream is reported drained. But the end-of-segment marker is the only way the sink learns that a stream has ended ("Indicates the end of a segment or the end of a stream" — Media Sinks › Markers).
4. The MPEG-4 sink interleaves the two streams by time. Once the longer stream runs about 1.3–1.4 s past the end of the shorter one, the sink stops issuing MEStreamSinkRequestSample for the longer stream, waiting for samples from the stream that has ended.
5. The longer stream therefore never gets the request that would place its marker and report it drained. video_finish_rx / audio_finish_rx never resolves, BeginFinalize is never called, and StopAndExportAsync waits forever. All unienc worker threads are idle at that point (parked in the thread pool), with nothing in WriteFile / WriteConsoleW or inside Media Foundation.

Whether an export hangs depends only on the end gap of the exported segment, so it is intermittent in real projects. In our project the video in the exported segment usually ends 0.6–1.2 s after the audio, and in about one run out of five 1.4–2.7 s after.

Changes

InstantReplay.Externals/unienc/crates/unienc_windows_mf/src/mux/mod.rs only.

• Pass a non-empty PROPVARIANT (VT_UI4) as pvarContextValue of the end-of-segment marker, so that the marker is accepted.
• Place the marker once per stream. The sink requests samples ahead of consuming them, so requests are still queued after the segment ends; those are now ignored instead of placing more markers, possibly on a sink that is already finalizing.
• Report the stream drained on MEStreamSinkMarker, i.e. once the sink has consumed everything placed before the marker, instead of right after placing it. This is the point from which finalization is safe to start.
• If PlaceMarker is still rejected, report the stream drained right away as before (no MEStreamSinkMarker follows a rejected marker).

The last three points are the same approach as 33ab159 on feature/windows-stability. That commit alone does not fix this hang, because the marker is still rejected and the fallback path behaves as before.

Verification

Standalone repro (no Unity)

A small C++ program drives MFCreateMPEG4MediaSink the same way unienc does (system-time presentation clock started and never stopped, one ProcessSample per MEStreamSinkRequestSample, PlaceMarker once the input is exhausted, BeginFinalize once both streams have drained). It uses real H.264 + AAC samples and trims one stream to vary the end gap. Source: mp4sink_repro_v5.cpp (attached).

mp4sink_repro_v5.cpp

Condition	NULL, NULL (current)	NULL, VT_UI4 (this PR)
Same length / 0.5 s end gap	OK	OK
3 s end gap, audio shorter	hang 3/3 (video stalls at 18.5 s / 20.1 s)	OK 3/3
3 s end gap, video shorter	hang 3/3	OK 3/3
Finalize after stopping the clock	same as above	OK
Total	—	30/30 OK

End-gap sweep with the current code (NULL, NULL), 5 runs per point:

End gap (longer − shorter)	audio shorter	video shorter
≤ 1.21 s	5/5 OK	5/5 OK
1.34 s	5/5 OK	0/5
1.44 – 1.66 s	2/5 OK	0/5
1.74 s	1/5 OK	0/5
≥ 1.85 s	0/5	0/5

So it hangs deterministically above about 1.3 s when video is the shorter stream, and with rising probability between about 1.4 and 1.85 s when audio is the shorter stream.

PlaceMarker argument variants with a 3 s gap: NULL/NULL, VT_EMPTY/VT_EMPTY, VT_EMPTY/NULL, NULL/VT_EMPTY → MF_E_INVALIDTYPE and hang. NULL/VT_UI4 → S_OK, MEStreamSinkMarker arrives, finalize OK.

For reference, IMFSinkWriter writing the same content with 3 s end gaps finalizes fine on the same machine.

In a Unity project (Unity 6000.2.13f1, Windows 11 24H2 / 10.0.26100.8875, RTX 4070, NVIDIA H.264 Encoder MFT)

RealtimeInstantReplaySession, 1280x720, fixed 15 fps, 2.5 Mbps, AAC with Wwise audio capture; 60 s recording, then StopAndExportAsync with a 120 s timeout.

Version	Result
v1.6.8	hang 2/2
v1.6.11	hang 2/12
v1.6.11 + this PR	OK 10/10, including runs with 1.87 s and 1.92 s end gaps, which hang without the fix

To see the end gap of runs that hang (their mp4 has no moov to read durations from), we measured the frames handed to the muxer: the export path was replicated step by step and the result of GetFramesForDuration was recorded before muxing. On successful runs this matches the durations in the exported mp4 within 0.03 s.

v1.6.11, 8 measured runs (video ends later than audio in every run):

Run	End gap (video − audio)	Result
1	0.64 s	OK
2	0.59 s	OK
3	0.67 s	OK
4	2.71 s	hang (2 PlaceMarker failed logs instead of 4)
5	0.88 s	OK
6	0.77 s	OK
7	0.70 s	OK
8	0.76 s	OK

Across all 18 measured runs (these 8 plus the 10 runs with this PR), the end gap was 0.56–1.17 s in 14 runs and 1.40, 1.87, 1.92 and 2.71 s in the other 4. Applying the thresholds from the standalone sweep, that gives an expected hang rate of about 15–20% for v1.6.11 in this setup, consistent with the 2/12 observed.

With this PR, PlaceMarker(ENDOFSEGMENT) failed (non-fatal) no longer appears in the log.

Notes

• pvarContextValue is only attached to the MEStreamSinkMarker event; its value is not used.
• The Windows build that rejects NULL context values has not been narrowed down beyond what Windows: export always fails with "Failed to complete muxer: channel closed" — OS MPEG4 sink rejects PlaceMarker (MF_E_INVALIDTYPE) #156 reports (26200, mfmp4srcsnk.dll 10.0.26100.8457). On builds that accept NULL, behavior is unchanged except that the stream is reported drained on MEStreamSinkMarker.
• Where the end gap comes from (separate issue, not changed here): in our runs the end gap closely follows the gap between the first and second video frame of the exported segment (e.g. 2.26 s in the run with a 2.71 s end gap, 0.55–0.75 s in the others). With StopAndExportAsync() and no duration, video starts at the first keyframe in the buffer and audio at the first audio frame, and each stream is rebased to 0 on its own. If the recording starts with a stall (e.g. scene loading right after the session is created), the video keeps the frame from before the stall while the audio effectively starts after it, so the video comes out longer by about the stall length. The same rebasing would also shift audio against video by that amount; we have not checked the exported files for this yet.
• Not changed here: the presentation clock is still never stopped before BeginFinalize. Media Session stops it first, but that is not required by the documentation, and it made no difference in the repro.