Bring PLaMo 3's YaRN range to the nearest whole dimensions on llama.cpp

README.md

@@ -472,8 +472,14 @@ Q4_K_M is the default because the model card reports Q4_0 0.90 chrF++ lower than
 Japanese and other languages, while both stay within 0.0011 of BF16 on its English–Japanese metrics.
 
 A BF16 GGUF converted from the original weights translated both fixtures byte-identically to the MLX FP16 backend.
-llama.cpp ignores the GGUF's `yarn_truncate = false` and rounds the YaRN correction range of the three full-attention
-layers; on the fixtures this changed FP32 logits by a mean KL divergence below 1e-4, far less than 4bit quantization.
+No llama.cpp release honors the GGUF's `yarn_truncate = false` yet
+([#30206](https://github.com/ggml-org/llama.cpp/pull/30206), [#30337](https://github.com/ggml-org/llama.cpp/pull/30337)):
+it widens the YaRN correction range of the three full-attention layers from 13.96–30.02 to 13–31. The error grows
+with input length, to a KL divergence of 0.04 at 2,048 tokens and 0.66 at 16,384, so the backend passes
+`--yarn-beta-fast` and `--yarn-beta-slow` values that make llama.cpp round to 14–30 instead, cutting it to 3e-4 and
+5e-3. In an 8,000-token document, 66 of 315 lines had differed from the MLX backend, including a mistranslated term;
+with the 14–30 range, 2 to 24 did ([summary](benchmarks/llama_cpp_plamo3_yarn_20261105/README.md)). These options
+must be removed once a llama.cpp release honors `yarn_truncate`.
 
 ### llama.cpp runtime and PLaMo 2 GGUFs
 

benchmarks/README.md

@@ -8,6 +8,7 @@ Raw output must stay in an ignored directory such as `.local/benchmarks/`.
 - [PLaMo 3 MLX measurements (default model)](plamo3_mlx_20261104/README.md)
 - [PLaMo 2 MLX measurements](../README.md#measured-result-2026-10-07)
 - [PLaMo 3 llama.cpp measurements](llama_cpp_plamo3_20261105/README.md)
+- [PLaMo 3 YaRN range on llama.cpp](llama_cpp_plamo3_yarn_20261105/README.md)
 - [PLaMo 2 llama.cpp measurements](llama_cpp_20261007/README.md)
 - [Standalone MLX release validation](releases/2026-10-07/README.md)
 

benchmarks/llama_cpp_plamo3_20261105/README.md

@@ -101,6 +101,11 @@ changed teacher-forced logits by a mean KL divergence of 9e-6 and 1e-5 on the fi
 sequence, with 100%, 100% and 99.91% top-1 agreement and reference NLL within 0.0014. 4bit quantization moves
 logits by a KL divergence of about 0.018.
 
+Correction: these short, partly repeated sequences hid the error, which grows with input length. Source-text
+predictions moved by a KL divergence of 0.04 at 2,048 tokens and 0.66 at 16,384, and an 8,000-token document
+translated differently in 66 of 315 lines. The backend now corrects most of it; see the
+[YaRN range summary](../llama_cpp_plamo3_yarn_20261105/README.md).
+
 chrF against the supplied references, English to Japanese / Japanese to English: BF16 GGUF 86.2 / 84.7,
 Q4_K_M 83.0 / 85.6, Q4_0 88.0 / 84.8, MLX 4bit 87.4 / 84.5. One document per direction cannot rank these. The model
 card's evaluation shows Q4_K_M within 0.02 chrF++ of BF16 on FLORES+ Japanese-multilingual pairs and Q4_0 0.90 lower,

benchmarks/llama_cpp_plamo3_yarn_20261105/README.md

@@ -0,0 +1,85 @@
+# PLaMo 3 YaRN range on llama.cpp, 2026-11-05
+
+Apple M1 Max, 64 GB. llama.cpp b11960 built with `scripts/build_llama_cpp.sh`, as in the
+[llama.cpp measurements](../llama_cpp_plamo3_20261105/README.md). Models: `pfnet/plamo-3-translate-2b` (original
+weights for MLX; BF16 GGUF converted with `scripts/prepare_llama_cpp.py`) and the official Q4_K_M GGUF.
+
+## Cause
+
+PLaMo 3 applies YaRN in its three full-attention layers (7, 15 and 23) and keeps the correction range fractional
+(`truncate: false`): dimension pairs 13.96 to 30.02 of 64 ramp from extrapolation to interpolation. No llama.cpp
+release honors the GGUF's `yarn_truncate = false` yet
+([#30206](https://github.com/ggml-org/llama.cpp/pull/30206) for ggml, CPU and CUDA only, and
+[#30337](https://github.com/ggml-org/llama.cpp/pull/30337) for PLaMo 3 are open): floor and ceil widen the range to
+13 to 31. The angle error of each changed pair grows linearly with the distance between tokens.
+
+The backend now passes `--yarn-beta-fast 31.738495 --yarn-beta-slow 1.0040927`, computed from the GGUF metadata, so
+that floor and ceil land on 14 and 30, the nearest whole dimensions. The measurements below used 31.676893 and
+1.0060454, which give the same 14 and 30. A llama.cpp that keeps the range fractional would use 14.001 to 29.999 with
+these options instead of the exact range, so they must go once a release honors `yarn_truncate`.
+
+## Logits, MLX FP32
+
+The FP32 MLX model with each range. Prompts are PFML3 English-to-Japanese translation requests whose source is English
+Markdown prose (this repository's README and llama.cpp's `docs`, without code blocks and tables), three documents per
+length. KL divergence from the exact range and top-1 agreement over the next-token distributions of the last 256
+prompt positions; medians over the three documents.
+
+| Prompt tokens | llama.cpp rounding (13–31) | With the options (14–30) | MLX 4bit, exact range |
+| --- | --- | --- | --- |
+| 545–598 | 1.8e-4 / 99.6% | 1.5e-7 / 100% | 0.16 / 78.9% |
+| 2,033–2,083 | 0.038 / 96.5% | 3.0e-4 / 100% | 0.41 / 69.5% |
+| 4,087–4,134 | 0.14 / 93.4% | 7.4e-4 / 99.6% | 0.51 / 70.7% |
+| 8,205–8,241 | 0.23 / 83.2% | 1.6e-3 / 98.8% | 0.53 / 70.3% |
+| 16,365–16,388 | 0.66 / 76.2% | 5.2e-3 / 98.0% | 0.43 / 72.3% |
+
+The first output token moved little (KL at most 0.06 with the rounding), and the earlier check on the fixtures'
+teacher-forced translations (mean KL at most 4e-5 up to 2,700 tokens) saw the same: confident predictions hide the
+error that source positions carry into later layers.
+
+## Logits, llama.cpp
+
+`llama-perplexity` on the same prose as plain text, against the BF16 GGUF with the options. KL divergence over the
+second half of each chunk: 4 chunks of 2,048 tokens, 2 of 8,192 and 1 of 16,384.
+
+| Scored positions | BF16, rounding | Q4_K_M, options | Q4_K_M, rounding (previous default) |
+| --- | --- | --- | --- |
+| 1,024–2,047 | 0.026 / 96.9% | 0.26 / 81.3% | 0.28 / 81.2% |
+| 4,096–8,191 | 0.30 / 86.2% | 0.53 / 74.8% | 0.67 / 72.3% |
+| 8,192–16,383 | 0.85 / 71.6% | 0.38 / 77.4% | 1.14 / 64.2% |
+
+Beyond about 8,000 tokens the rounding moves predictions more than Q4_K_M quantization does.
+
+## Translations
+
+Greedy English-to-Japanese translations of one document per length (the first of the three above), compared with the
+fused MLX FP16 backend, which uses the exact range. Lines that differ from it, by `difflib`:
+
+| Source tokens | Lines | MLX FP16, 13–31 | MLX FP16, 14–30 | llama-server BF16, rounding | llama-server BF16, options |
+| ---: | ---: | ---: | ---: | ---: | ---: |
+| 2,064 | 75 | 9 | 1 | 9 | 1 |
+| 4,115 | 208 | 17 | 14 | 18 | 9 |
+| 8,186 | 315 | 66 | 24 | 66 | 2 |
+
+chrF of `llama-server`'s output against MLX rose from 98.4, 98.6 and 96.5 to 99.8, 99.4 and 99.9 with the options.
+Greedy outputs diverge after the first changed token, and with the same 14–30 range MLX and `llama-server` split at
+character 7,724 of the 8,186-token document, so the gain varies between runs: 2 or 24 lines there.
+
+Applying llama.cpp's rounding to the MLX model reproduced `llama-server`'s 8,186-token translation byte for byte; at
+2,064 and 4,115 tokens the two agreed for the first 4,393 and 5,791 characters. Most differences from the exact range
+were wording, such as グループ化 against 集約, but the rounding also translated "zero in-loop vector spills" as
+ループ内量子化がゼロ ("zero in-loop quantization") and dropped the parenthetical "of 4 `k` each" (各グループ4個). No line was
+dropped.
+
+With the options, the BF16 GGUF still translated both fixtures byte-identically to MLX FP16, and Q4_K_M decoded the
+English fixture at 112 tok/s (111 before) with the same output.
+
+## Limitations
+
+- Only M1 Max and English source text were measured, with one generated document per length, so the translation
+  counts are examples rather than rates.
+- Predicting Markdown prose is less certain than translating it, so every KL divergence here, including quantization,
+  is larger than on translation outputs.
+- The options approximate the exact range, and their error also grows with length. An exact range needs the upstream
+  changes, and then these options must be removed; #30206 has no Metal kernel, so on Apple Silicon the operation would
+  fall back to the CPU (not measured).

src/plamo_translate/servers/llama_cpp/model.py

@@ -82,6 +82,15 @@ def same_model(first: str | None, second: str | None) -> bool:
 
 def gguf_architecture(path: str) -> str:
     """`general.architecture` from a GGUF header, read without loading any tensor."""
+    metadata = gguf_metadata(path, until="general.architecture")
+    if not isinstance(metadata.get("general.architecture"), str):
+        raise ValueError(f"{path} does not declare general.architecture")
+    return metadata["general.architecture"]
+
+
+def gguf_metadata(path: str, until: str | None = None) -> dict:
+    """Scalar and string values of a GGUF header, up to and including the key `until`; arrays are skipped."""
+    metadata = {}
     with open(path, "rb") as stream:
 
         def read(form: str):
@@ -117,8 +126,14 @@ def skip(kind: int) -> None:
             raise ValueError(f"{path} is not a GGUF version 2 or 3 file")
         read("Q")  # tensor count
         for _ in range(read("Q")):
-            key, kind = string(), read("I")
-            if key == b"general.architecture" and kind == _STRING:
-                return string().decode("utf-8")
-            skip(kind)
-    raise ValueError(f"{path} does not declare general.architecture")
+            # llama.cpp does not require UTF-8 strings, so neither does this reader.
+            key, kind = string().decode("utf-8", "replace"), read("I")
+            if kind in _SCALARS:
+                metadata[key] = read(_SCALARS[kind])
+            elif kind == _STRING:
+                metadata[key] = string().decode("utf-8", "replace")
+            else:
+                skip(kind)
+            if key == until:
+                break
+    return metadata

src/plamo_translate/servers/llama_cpp/server.py

@@ -2,6 +2,7 @@
 
 import asyncio
 import json
+import math
 import os
 import platform
 import re
@@ -17,7 +18,7 @@
 import httpx
 from mcp.server.fastmcp import Context, FastMCP
 
-from plamo_translate.servers.llama_cpp.model import SUPPORTED_ARCHITECTURES, gguf_architecture
+from plamo_translate.servers.llama_cpp.model import SUPPORTED_ARCHITECTURES, gguf_architecture, gguf_metadata
 from plamo_translate.servers.mlx.pfml3 import pfml3_prompt
 from plamo_translate.servers.utils import (
     INSTRUCTION,
@@ -48,6 +49,37 @@ def gguf_file(name: str, option: str) -> str:
     return str(path)
 
 
+def yarn_range_arguments(metadata: dict) -> list[str]:
+    """llama-server options that bring a fractional YaRN correction range to its nearest whole dimensions.
+
+    PLaMo 3 ramps between fractional dimensions (`truncate: false`; 13.96 to 30.02 for the 2B models), but no
+    llama.cpp release honors that yet (ggml-org/llama.cpp#30206, #30337): floor and ceil widen the range to 13 to 31.
+    The angle error grows with the distance between tokens: on source text, the KL divergence from the exact range
+    reached 0.04 at 2,048 tokens and 0.66 at 16,384. Betas whose dimensions lie just inside the nearest whole numbers
+    make floor and ceil land on 14 and 30, cutting it to 3e-4 and 5e-3. A llama.cpp with those changes would use
+    14.001 to 29.999 instead of the exact range, so drop these options once a release honors `yarn_truncate`.
+    """
+    architecture = metadata.get("general.architecture")
+    scaling = f"{architecture}.rope.scaling."
+    if metadata.get(scaling + "type") != "yarn" or metadata.get(scaling + "yarn_truncate", True):
+        return []
+    dimensions = metadata.get(f"{architecture}.rope.dimension_count") or metadata.get(
+        f"{architecture}.attention.key_length"
+    )
+    original = metadata.get(scaling + "original_context_length")
+    base = metadata.get(f"{architecture}.rope.freq_base")
+    if not dimensions or not original or not base:
+        return []
+    # ggml_rope_yarn_corr_dim and its inverse.
+    scale = dimensions / (2 * math.log(base))
+    low = round(scale * math.log(original / (metadata.get(scaling + "yarn_beta_fast", 32.0) * 2 * math.pi))) + 0.001
+    high = round(scale * math.log(original / (metadata.get(scaling + "yarn_beta_slow", 1.0) * 2 * math.pi))) - 0.001
+    if not 0 < low < high:
+        return []
+    beta = lambda dimension: original / (2 * math.pi * math.exp(dimension / scale))  # noqa: E731
+    return ["--yarn-beta-fast", f"{beta(low):.8g}", "--yarn-beta-slow", f"{beta(high):.8g}"]
+
+
 @dataclass(frozen=True)
 class LlamaCppOptions:
     model: str
@@ -130,6 +162,7 @@ def command(self, port: int) -> list[str]:
             # The n_rot line checked by validate_model_log is logged at this verbosity.
             "--log-verbosity",
             "4",
+            *yarn_range_arguments(gguf_metadata(model)),
             *speculation,
         ]
 

tests/test_llama_cpp.py

@@ -1,5 +1,6 @@
 import asyncio
 import json
+import math
 import os
 import struct
 import subprocess
@@ -19,6 +20,7 @@
     DEFAULT_GGUF_MODEL,
     canonical_model,
     gguf_architecture,
+    gguf_metadata,
     resolve_model,
     same_model,
 )
@@ -30,6 +32,7 @@
     completion_events,
     completion_payload,
     validate_model_log,
+    yarn_range_arguments,
 )
 from plamo_translate.servers.utils import Message, TranslateRequest
 
@@ -59,14 +62,30 @@ def gguf_string(text):
     return struct.pack("<Q", len(data)) + data
 
 
-def write_gguf(path, architecture="plamo3", before=()):
-    """A header-only GGUF; `before` holds (key, value type, encoded value) entries placed ahead of the architecture."""
-    entries = [*before, ("general.architecture", 8, gguf_string(architecture))]
+def write_gguf(path, architecture="plamo3", before=(), after=()):
+    """A header-only GGUF; `before` and `after` hold (key, value type, encoded value) entries around the architecture."""
+    entries = [*before, ("general.architecture", 8, gguf_string(architecture)), *after]
     body = b"".join(gguf_string(key) + struct.pack("<I", kind) + value for key, kind, value in entries)
     path.write_bytes(b"GGUF" + struct.pack("<IQQ", 3, 0, len(entries)) + body)
     return path
 
 
+def plamo3_rope(truncate=False, beta_fast=32, beta_slow=1, dimension_count=None):
+    """The RoPE entries of the official PLaMo 3 Translate GGUFs, unless overridden."""
+    entries = [
+        ("plamo3.rope.scaling.type", 8, gguf_string("yarn")),
+        ("plamo3.rope.scaling.factor", 6, struct.pack("<f", 256)),
+        ("plamo3.rope.scaling.original_context_length", 4, struct.pack("<I", 4096)),
+        ("plamo3.rope.scaling.yarn_beta_fast", 6, struct.pack("<f", beta_fast)),
+        ("plamo3.rope.scaling.yarn_beta_slow", 6, struct.pack("<f", beta_slow)),
+        ("plamo3.rope.freq_base", 6, struct.pack("<f", 1_000_000)),
+        ("plamo3.attention.key_length", 4, struct.pack("<I", 128)),
+    ]
+    if dimension_count is not None:
+        entries.append(("plamo3.rope.dimension_count", 4, struct.pack("<I", dimension_count)))
+    return [*entries, ("plamo3.rope.scaling.yarn_truncate", 7, bytes([truncate]))]
+
+
 def test_prompt_preserves_template_and_does_not_mutate_request():
     original = request()
     expected = (
@@ -137,6 +156,69 @@ def test_gguf_architecture_skips_every_value_type(tmp_path):
     assert gguf_architecture(str(write_gguf(tmp_path / "plamo2.gguf", "plamo2"))) == "plamo2"
 
 
+def test_gguf_architecture_stops_at_the_architecture(tmp_path):
+    # Entries after the architecture are not read: an unknown value type there does not matter.
+    path = write_gguf(tmp_path / "model.gguf", after=[("future", 99, b"")])
+    assert gguf_architecture(str(path)) == "plamo3"
+    with pytest.raises(ValueError, match="unknown GGUF value type"):
+        gguf_metadata(str(path))
+
+
+def test_gguf_metadata_accepts_strings_that_are_not_utf8(tmp_path):
+    # llama.cpp loads such files, so reading the YaRN keys must not stop them.
+    after = [("general.name", 8, struct.pack("<Q", 1) + b"\xff"), *plamo3_rope()]
+    metadata = gguf_metadata(str(write_gguf(tmp_path / "model.gguf", after=after)))
+    assert metadata["general.name"] == "\ufffd"
+    assert yarn_range_arguments(metadata)
+
+
+def test_gguf_metadata_reads_scalars_and_strings(tmp_path):
+    before = [*plamo3_rope(), ("tokens", 9, struct.pack("<IQ", 8, 1) + gguf_string("<|plamo:tag|>"))]
+    metadata = gguf_metadata(str(write_gguf(tmp_path / "model.gguf", before=before)))
+    assert metadata["general.architecture"] == "plamo3"
+    assert metadata["plamo3.rope.scaling.type"] == "yarn"
+    assert metadata["plamo3.rope.scaling.original_context_length"] == 4096
+    assert metadata["plamo3.rope.freq_base"] == 1_000_000
+    assert metadata["plamo3.rope.scaling.yarn_truncate"] is False
+    assert "tokens" not in metadata
+
+
+def corr_dim(beta, dimensions=128):
+    """ggml_rope_yarn_corr_dim for the RoPE of the official PLaMo 3 Translate GGUFs."""
+    return dimensions * math.log(4096 / (beta * 2 * math.pi)) / (2 * math.log(1_000_000))
+
+
+@pytest.mark.parametrize(
+    "rope,dimensions,betas",
+    [
+        # The official GGUFs: llama.cpp widens the exact 13.96..30.02 to 13..31.
+        ({}, 128, (32, 1)),
+        # Other betas and a rotary dimension count below the head size must be read from the GGUF.
+        ({"beta_fast": 16, "beta_slow": 2, "dimension_count": 64}, 64, (16, 2)),
+    ],
+)
+def test_yarn_range_moves_to_nearest_whole_dimensions(tmp_path, rope, dimensions, betas):
+    metadata = gguf_metadata(str(write_gguf(tmp_path / "model.gguf", before=plamo3_rope(**rope))))
+    arguments = yarn_range_arguments(metadata)
+    options = dict(zip(arguments[::2], arguments[1::2]))
+    low = corr_dim(float(options["--yarn-beta-fast"]), dimensions)
+    high = corr_dim(float(options["--yarn-beta-slow"]), dimensions)
+    exact = corr_dim(betas[0], dimensions), corr_dim(betas[1], dimensions)
+    assert (math.floor(exact[0]), math.ceil(exact[1])) != (round(exact[0]), round(exact[1]))
+    # llama.cpp applies floor and ceil; with these betas they land on the nearest whole dimensions.
+    assert (math.floor(low), math.ceil(high)) == (round(exact[0]), round(exact[1]))
+    # A llama.cpp that keeps the range fractional would use the same dimensions within 0.001.
+    assert abs(low - round(exact[0])) < 0.0011 and abs(high - round(exact[1])) < 0.0011
+
+
+def test_yarn_range_is_left_alone_unless_the_gguf_keeps_it_fractional(tmp_path):
+    truncated = gguf_metadata(str(write_gguf(tmp_path / "truncated.gguf", before=plamo3_rope(truncate=True))))
+    assert yarn_range_arguments(truncated) == []
+    unspecified = gguf_metadata(str(write_gguf(tmp_path / "unspecified.gguf", before=plamo3_rope()[:-1])))
+    assert yarn_range_arguments(unspecified) == []
+    assert yarn_range_arguments(gguf_metadata(str(write_gguf(tmp_path / "plamo2.gguf", "plamo2")))) == []
+
+
 @pytest.mark.parametrize("content", [b"", b"GGML" + bytes(20), b"GGUF" + struct.pack("<IQQ", 1, 0, 0)])
 def test_gguf_architecture_rejects_other_files(tmp_path, content):
     path = tmp_path / "model.gguf"
@@ -248,6 +330,10 @@ def test_command_tunes_server_and_adds_draft(monkeypatch, tmp_path):
     assert command[command.index("--model-draft") + 1] == str(draft)
     assert command[command.index("--spec-type") + 1] == "draft-simple"
     assert command[command.index("--gpu-layers-draft") + 1] == "99"
+    assert "--yarn-beta-fast" not in command
+    yarn = LlamaCppOptions(str(write_gguf(tmp_path / "yarn.gguf", before=plamo3_rope())), sys.executable).command(8080)
+    assert yarn[yarn.index("--yarn-beta-fast") + 1] == "31.738495"
+    assert yarn[yarn.index("--yarn-beta-slow") + 1] == "1.0040927"
     cpu = LlamaCppOptions(str(model), sys.executable, gpu_layers=0).command(8080)
     assert cpu[cpu.index("--threads") + 1] == "8"
     assert "--model-draft" not in cpu