Skip to content

FIX: long-term reference list reordering in non-Baseline CABAC streams - #3954

Open
niw wants to merge 2 commits into
cisco:masterfrom
niw:fix_long_term_references
Open

niw wants to merge 2 commits into
cisco:masterfrom
niw:fix_long_term_references

Conversation

@niw

@niw niw commented Jun 17, 2026

Copy link
Copy Markdown

Problem

When a slice carries ref_pic_list_reordering commands for long-term references (reordering_of_pic_nums_idc == 2), WelsReorderRefList2() matched the requested long_term_pic_num against each picture's uiLongTermPicNum field.

That field is unreliable because long-term references are created via MMCO 3 (short-to-long) and MMCO 6 (mark current as long), both of which only carry long_term_frame_idx in the bitstream and never a long_term_pic_num. As a result AddLongTermToList() stores uiLongTermPicNum as 0 for every such picture.

With multiple long-term references present, they all compare equal (0), so the reorder failed to select the requested entry (matching either the wrong one or none at all), leaving the wrong picture at the head of the reference list and corrupting inter prediction.

It only surfaces when multiple long-term references and a long-term reorder command occur together. In practice this is hit when decoding AirPlay Mirroring streams from iOS and macOS, whose High-profile CABAC streams use multiple long-term references with reordering, producing visibly corrupted video.

Note that WelsReorderRefList2() is only used for non-Baseline CABAC streams (uiProfileIdc != 66 && bEntropyCodingModeFlag).

The problem was introduced disabled (#if 0) by 1b8caef, and later enabled by fa6d099, which flipped it to #if 1 and exposed it.

Solution

Match by iLongTermFrameIdx instead of uiLongTermPicNum, in both the selection loop and the subsequent compaction loop.

iLongTermFrameIdx is the identifier that is actually maintained on the picture (set in AddLongTermToList) and, per the spec (8.2.4.1), for frame coding LongTermPicNum == LongTermFrameIdx, so the value parsed as long_term_pic_num matches it directly.

This aligns the reorder path with how the rest of the code already identifies long-term pictures, e.g. WelsDelLongFromList() used by MMCO 2, which likewise matches against iLongTermFrameIdx.

Testing

Added a focused unit test (DecUT_RefListReorder.cpp) that reproduces the case via three long-term references sharing uiLongTermPicNum == 0 with distinct iLongTermFrameIdx, plus an example High-profile CABAC regression stream (res/ltr_reorder_high_cabac.264) wired into the decoder API test.

Considerations

Keeping the match on uiLongTermPicNum but populating it correctly, i.e. deriving LongTermPicNum from LongTermFrameIdx per slice (8.2.4.1) or setting it at marking time, would be an alternative option. However, LongTermPicNum is a per-slice derived quantity, not a persistent property of the picture, so storing it is spec-awkward. For frame coding, it just equals LongTermFrameIdx, making the field redundant.
It would also touch the marking path (AddLongTermToList, MarkAsLongTerm) for no functional gain.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The unit-test fixture records an empty active reference list, causing the reordered tail to be overwritten and masking malformed output.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Fixes long-term reference reordering in non-Baseline CABAC decoding by matching the maintained frame index.

Changes:

  • Matches and compacts long-term references using iLongTermFrameIdx.
  • Adds unit and stream-level regression coverage.
  • Registers the new unit test across build systems.
File Description
codec/​decoder/​core/​src/​manage_dec_ref.cpp Corrects long-term reference matching.
test/​decoder/​DecUT_RefListReorder.cpp Adds focused reorder coverage.
test/​decoder/​targets.mk Registers the unit test for Make.
test/​decoder/​meson.build Registers the unit test for Meson.
test/​build/​win32/​codec_ut/​codec_unittest.vcproj Registers the unit test for Windows.
test/​api/​decoder_test.cpp Adds the regression stream and output hash.
res/​ltr_reorder_high_cabac.264 Provides the CABAC regression stream.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/decoder/DecUT_RefListReorder.cpp Resolved
@niw
niw force-pushed the fix_long_term_references branch from 4517797 to 52bd2c3 Compare September 24, 2026 04:40
@eduar-hte

Copy link
Copy Markdown

We hit the same bug decoding the H.264 that Windows Remote Desktop sends (the RDP graphics pipeline's AVC444 mode, MS-RDPEGFX), through FreeRDP's OpenH264 backend, and can confirm this PR fixes it.

The scenario. In AVC444v2 the server sends each frame as two pictures, a regular 4:2:0 one (luma) and an auxiliary one carrying the remaining chroma samples, interleaved in a single H.264 sequence and decoded by one decoder instance (h264_openh264.c in FreeRDP). The Windows encoders we've seen keep one long-term reference per view:

  • Main profile (profile_idc 77), CABAC, max_num_ref_frames 3, pic_order_cnt_type 2.
  • After each IDR, an MMCO 4 (max_long_term_frame_idx_plus1 2); then every reference picture marks itself long-term with MMCO 6: luma pictures long_term_frame_idx 0, chroma pictures 1.
  • A P slice that predicts from its own view's previous picture moves it to the front of list 0: modification_of_pic_nums_idc 2, long_term_pic_num 0 or 1.

As both long-term pictures have uiLongTermPicNum 0 (MMCO 6 never sets it), WelsReorderRefList2 can't tell them apart, and a slice asking for long_term_pic_num 1 is predicted from the other view's picture. Nothing reports an error (dsErrorFree): the chroma of those frames is just wrong.

When it was introduced. In v2.5.0, by fa6d099 (#3723), which switched InitRefPicList's #if 0 to #if 1 and so sent non-Baseline CABAC streams through WelsReorderRefList2. v2.4.1 decodes these streams correctly, so we've stayed on it since.

Measured. Native builds of each version. Every access unit was decoded as FreeRDP does (DecodeFrame2, then FlushFrame), and every picture's I420 planes were compared with ffmpeg's decode of the same stream (-f framecrc):

Windows RDP AVC444v2 stream v2.4.1 v2.6.0 v2.7.0 v2.7.0 + this PR
A, 115 pictures exact 7 wrong 7 wrong exact
B, 58 pictures exact 8 wrong 8 wrong exact
C, 63 pictures (12 slices reorder to long_term_pic_num 1) exact 12 wrong 12 wrong exact
D, 1,381 pictures exact 63 wrong 63 wrong exact

We also checked a stream from another Windows host whose encoder marks the chroma pictures long_term_frame_idx 1 without the MMCO 4: v2.7.0 with this PR decodes all of its 5,069 pictures exactly too.

@eduar-hte

Copy link
Copy Markdown

A follow-up: a correction to my description above, and a 3-picture stream that reproduces the bug.

Correction. It isn't only the chroma pictures. The Windows encoders in our captures mark every picture long-term and reorder by long_term_pic_num. Right after an IDR, the first pictures, all from the main 4:2:0 view, already use both long-term indexes:

Picture Slice ref_pic_list_modification (list 0) dec_ref_pic_marking
1 IDR – long_term_reference_flag 1 (index 0)
2 P, frame_num 1 long_term_pic_num 0 MMCO 4 (max_long_term_frame_idx_plus1 2), MMCO 6 to index 1
3 P, frame_num 2 long_term_pic_num 1 MMCO 6 to index 0

Picture 3 asks for long-term picture 1 (picture 2), and v2.5.0 to v2.7.0 predict it from picture 1. The split I described (main view on index 0, chroma on index 1) only comes after that. So all 8 wrong pictures of stream B are main-view ones, and 3 of the 7 in A. In C and D, all of them are chroma.

A 3-picture stream. The first three pictures of one of B's sequences, unmodified (attached: rdp_ltr_reorder_3_pictures.264.zip). It's 93,027 bytes: Main profile, level 3.1, CABAC, 1056x752, two slices per picture, max_num_ref_frames 3, pic_order_cnt_type 2. Each tree's h264dec, against ffmpeg:

Decoder Pictures 1 and 2 Picture 3 SHA-1 of the output
ffmpeg 8.1.1 eae586cfdbff4fc9aef07090cdc16502107260f4
v2.4.1 as ffmpeg as ffmpeg eae586cf…
v2.6.0, v2.7.0 as ffmpeg 371,732 of 1,191,168 samples differ (by up to 31) e42fd63db038206af8c9ad1401ba4f71a96f3271
v2.7.0 + this PR as ffmpeg as ffmpeg eae586cf…

The SHA-1 covers the whole I420 output, row by row, the way test/api/decoder_test.cpp hashes it. If a stream from a production encoder helps as a second case next to res/ltr_reorder_high_cabac.264, it can go in kFileParamArray as is.

niw added 2 commits October 30, 2026 08:13
**Problem**

When a slice carries `ref_pic_list_reordering` commands for long-term
references (`reordering_of_pic_nums_idc == 2`), `WelsReorderRefList2()`
matched the requested `long_term_pic_num` against each picture's
`uiLongTermPicNum` field.

That field is unreliable because long-term references are created via
MMCO 3 (short-to-long) and MMCO 6 (mark current as long),
both of which only carry `long_term_frame_idx` in the bitstream and
never a `long_term_pic_num`. As a result `AddLongTermToList()` stores
`uiLongTermPicNum` as `0` for every such picture.

With multiple long-term references present, they all compare equal
(`0`), so the reorder failed to select the requested entry (matching
either the wrong one or none at all), leaving the wrong picture at the
head of the reference list and corrupting inter prediction.

It only surfaces when multiple long-term references and a long-term
reorder command occur together. In practice this is hit when decoding
AirPlay Mirroring streams from iOS and macOS, whose High-profile CABAC
streams use multiple long-term references with reordering, producing
visibly corrupted video.

Note that `WelsReorderRefList2()` is only used for non-Baseline CABAC
streams (`uiProfileIdc != 66 && bEntropyCodingModeFlag`).

The problem was introduced disabled (`#if 0`) by 1b8caef, and later
enabled by fa6d099, which flipped it to `#if 1` and exposed it.

**Solution**

Match by `iLongTermFrameIdx` instead of `uiLongTermPicNum`, in both the
selection loop and the subsequent compaction loop.

`iLongTermFrameIdx` is the identifier that is actually maintained
on the picture (set in `AddLongTermToList`) and, per the spec (8.2.4.1),
for frame coding `LongTermPicNum == LongTermFrameIdx`, so the value
parsed as `long_term_pic_num` matches it directly.

This aligns the reorder path with how the rest of the code already
identifies long-term pictures, e.g. `WelsDelLongFromList()`
used by MMCO 2, which likewise matches against `iLongTermFrameIdx`.

**Testing**

Added a focused unit test (`DecUT_RefListReorder.cpp`) that reproduces
the case via three long-term references sharing `uiLongTermPicNum == 0`
with distinct `iLongTermFrameIdx`, plus an example High-profile CABAC
regression stream (`res/ltr_reorder_high_cabac.264`) wired into the
decoder API test.

**Considerations**

Keeping the match on `uiLongTermPicNum` but populating it correctly,
i.e. deriving `LongTermPicNum` from `LongTermFrameIdx` per slice
(8.2.4.1) or setting it at marking time, would be an alternative option.
However, `LongTermPicNum` is a per-slice derived quantity,
not a persistent property of the picture, so storing it is spec-awkward.
For frame coding, it just equals `LongTermFrameIdx`, making the field
redundant.
It would also touch the marking path (`AddLongTermToList`,
`MarkAsLongTerm`) for no functional gain.
@niw
niw force-pushed the fix_long_term_references branch from 52bd2c3 to fccad3d Compare October 30, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants