Skip to content

orchestration, pis: move the gwas catalog top hits release into a dated prod run folder - #220

Merged
project-defiant merged 1 commit into
mainfrom
chore/gwas-catalog-top-hits-prod-run
Oct 28, 2026
Merged

project-defiant merged 1 commit into
mainfrom
chore/gwas-catalog-top-hits-prod-run

Conversation

@project-defiant

@project-defiant project-defiant commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The 2026-08-28 GWAS Catalog top hits run was moved from the root of gs://gwas_catalog_top_hits into a dated run folder, so the release is preserved, before we run the v4.0.0 gentropy release with updated credible set schema.
This will allow us to compare the results and roll back in case of hard issues discovered upon release.

gs://gwas_catalog_top_hits/prod/20260828T165117/
├── config/gwas_catalog_top_hits.yaml   # DAG config of the run
├── docs/                               # README.md + DAG svg
└── output/{study_index, study_locus_window_based_clumped, study_locus_ld_clumped, credible_sets}

This PR points the repo at the new location:

  • gwas_catalog_top_hits.yaml: output_path is now gs://gwas_catalog_top_hits/prod/20260828T165117/output. It is identical to the config uploaded to the run folder.
  • pis/config.yaml: the GCCA study index and credible sets are copied from the run's output/.
  • Docs: the top hits section moves to docs/datasources/gwas_catalog_data/gwas_catalog_top_hits/README.md. The new README:
    • documents the inputs and how gentroutils syncs them from the EBI FTP
    • links gentropy at v3.4.0-rc.3, the version in the config
    • has schema tables of the four outputs, with a description per field
    • The DAG svg is regenerated: its edges pointed at cluster task ids that did not match the nodes, so it rendered broken. The GWAS Catalog README links to the new page and gets a changelog entry.
  • Makefile: upload-gwas-catalog-top-hits-readme uploads only the top hits docs to the run folder (GWAS_CATALOG_TOP_HITS_RUN), and is part of update-bucket-docs. The GWAS Catalog upload no longer copies the full docs to the top hits bucket.

The bucket is already in this state. The data, config and docs were moved and uploaded, and the files in this PR are identical to the uploaded copies.

Notes

  • With output_path pinned to the run folder, running the old top hits DAG again would overwrite this release. Bump the timestamp before the next run.

Test plan

  • uv run pytest tests/test_dag_validation.py (orchestration): 10 passed
  • uv run pytest tests (pis): 10 passed
  • make -n upload-gwas-catalog-top-hits-readme resolves to the run folder
  • README, svg and config in the PR are byte-identical to gs://gwas_catalog_top_hits/prod/20260828T165117/
  • study_index/*.parquet and credible_sets/*.parquet globs in pis/config.yaml match files in the run folder

🤖 Generated with Claude Code

…ed prod run folder

The 2026-08-28 top hits run now lives in
gs://gwas_catalog_top_hits/prod/20260828T165117/ with its outputs under
output/, its DAG config under config/ and its docs under docs/.

- gwas_catalog_top_hits.yaml: output_path points at the run's output/ folder
- pis/config.yaml: GCCA study index and credible sets read from the run folder
- docs: split the top hits docs into gwas_catalog_top_hits/README.md, with
  the inputs, the gentroutils sync, gentropy links pinned to v3.4.0-rc.3 and
  the output schemas; regenerate the broken DAG svg
- Makefile: upload-gwas-catalog-top-hits-readme uploads only the top hits
  docs to the run folder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@project-defiant
project-defiant merged commit ef0463b into main Oct 28, 2026
10 checks passed
@project-defiant project-defiant linked an issue Oct 29, 2026 that may be closed by this pull request
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Validate migrated credible-set sources against their frozen releases Staging data freeze pre-migration

2 participants