[DSuveges]

Summary

Carries over the schema and validation logic from #111, which is otherwise obsolete. Output datasets are declared once as Pydantic models. The pandera schema used for validation, and optional documentation JSON, are derived from them. The JSON dumps themselves are not committed.

pts/src/pts/schemas/
├── dataset/           # DatasetModel/StructModel, field tree (spec), pandera converter, checks, discovery, docs generator
└── outputs/           # evidence_base, gwas_evidence, encore_evidence, biosample

Validation is wired into these three steps:

• write_dataset(..., schema=Model): checks the destination first, then validates the frame and writes it. A violation writes nothing. This is used by biosample.
• EvidenceDatasource(schema=...), passed through run_evidence to write_evidence. This is used by gwas_credible_sets and encore. Excluded evidence and alongside sinks are streamed to disk unvalidated, in the same pass that collects the valid evidence. Only the valid part is held in memory and validated, and the input is still computed once. If validation fails, failed_evidence is written but evidence is not.

Changes compared with #111:

• The models are checked against what main's transformers write now. GWAS and ENCORE carry over unchanged, apart from module and step references. The quality-flag vocabulary is tested against EvidenceFlags.
• validationReadouts is added to evidence_schema. Harmonisation then fixes its struct field order and its hsaValue dtype, which pandera checks but Pydantic does not.
• Models that override model_config, and fields using validate_default, are rejected. Pydantic would enforce them, but the derived schema would not.
• The JSON docs under pts/docs/schemas/ and the test that checked they were up to date are dropped. The generator is kept: uv run python -m pts.schemas.dataset.docs <dir>.
• The GWAS and ENCORE transformer tests now use realistic fixtures (Ensembl-shaped ids, complete ENCORE rows), because placeholder ids no longer pass validation.

New dependency: pandera[polars].

Test run on 26.09 data

All three steps were run end to end on local copies of run vh/ppp-2609-1's inputs, with validation enabled:

Step	Valid rows (this branch / release)	Failed rows (this branch / release)	ids and columns match
biosample	35,477 / 35,477	—	same ids*
encore	112,134 / 112,134	794 / 794	yes
gwas_credible_sets	3,467,408 / 3,467,408	0 / 0	yes

The release outputs of all three also pass their models.

* Biosample content differs from the release in ways that predate this PR (main's port): empty lists instead of nulls for parents/children, 63 labels instead of purl URLs, and children/descendants swapped in column order.

Benchmark: with vs without validation

Step	Without	With	Peak memory
biosample (median of 5)	7.94 s	7.93 s	—
encore (median of 3)	0.3 s	0.3 s	—
gwas_credible_sets (median of 3, one process)	7.1 s	7.5 s	—
gwas_credible_sets (separate processes, 2 runs each)	8.6–8.7 s	8.6–9.4 s	4.9–5.6 GiB either way

On 3.5M GWAS rows, validation costs about 0.4 s. Peak memory is dominated by the literature look-up table, not by the validated evidence. On synthetic GWAS evidence, validation took 0.09 s for 1M rows and 0.41 s for 5M rows.

Known limits

• Pandera validates the collected valid output, so a dataset with tens of millions of valid rows would hold them in memory. A streaming alternative is possible: evaluate the converter's per-rule expressions as violation counts in the same pass as the sinks. The cost is that violations would only be found after writing. It is not built.
• NaN is treated as null by the derived schema. This deliberately deviates from Pydantic and is documented in the converter and the agreement test.

Test plan

• make lint-pts
• make test-pts (2882 passed, 11 skipped)
• End-to-end runs on 26.09 data, as above

🤖 Generated with Claude Code