Product bundles by mikegottlieb84 · Pull Request #40 · EarthScope/GNSSommelier

mikegottlieb84 · 2026-09-04T16:26:50Z

Summary

This PR improves PRIDE PPP-AR product selection, same-day processing, failure handling, diagnostics, and output preservation.

It introduces coherent precise-product bundles so orbit, clock, ERP, bias, and optional attitude products are selected from compatible analysis-center/product families rather than independently mixing incompatible products.

Product bundle resolution

Added dependency bundles with configurable coherence fields.
PRIDE precise products are grouped by:
- Analysis center (AAA)
- Timeliness (TTT)
- Product line (PPP)
Bundle candidates are preflighted before downloading.
Required bundle members must exist within the same family.
If a required download fails, the complete family is rejected and the next compatible family is tried.
Product searches and downloads within a bundle run concurrently.
Independent dependencies, such as broadcast navigation and static tables, continue resolving separately.
Added optional analysis-center restrictions for targeted testing and diagnostics.
Aggregate lockfiles are only written when all required dependencies are fulfilled.

Product precedence and sources

Default PRIDE precedence is now:
1. FIN
2. RTS
3. RAP
4. ULT
Added Wuhan RTS orbit, 5-second clock, ERP, and OSB support.
RTS is preferred over RAP because the Wuhan RTS bias products support the configured Galileo E1/E5b combination.
Updated CODE products from retired FTP access to the current AIUB HTTPS/S3-backed archive.
Added correct AIUB directory-listing support.
Added BKG’s rolling WRD current-day mixed-GNSS broadcast navigation product.
Broadcast navigation preference is now WRD, followed by normal IGS/IGN/DLR daily products.
Added coherent product bundles to both default and final-only PRIDE dependency specifications.

Download and cache behavior

Added force_download support throughout dependency resolution and transport.
override_products_download=True now refreshes mutable products that can change, such as orbit, clock, bias, ERP, navigation, and attitude products.
Static PRIDE tables use refresh_on_force: false and remain cached.
Cached decompressed products are bypassed when a fresh download is explicitly requested.
Sidecar lockfiles are refreshed after forced downloads.
Added one bounded retry with randomized backoff for transient network failures.
Failed remote downloads are memoized for the duration of a resolution run so the same missing URL is not repeatedly retried.
Added safer reconnection behavior for stale FTP/HTTP connections.
GitHub-hosted files larger than the Contents API limit are downloaded through immutable raw-content URLs.
Product metadata is classified from the actual matched filename, allowing bundle grouping to use concrete family values rather than wildcard search patterns.
Remote URI construction now avoids duplicated schemes and path separators.

PRIDE product compatibility diagnostics

Added advisory validation before processing:

Reads the phase observables advertised by each RINEX file.
Reads the phase observables available in Bias-SINEX products.
Reports signal-level and frequency-band bias coverage by constellation.
Checks SP3 orbit and RINEX clock epoch coverage against the observation interval.
Reports partial constellation or time coverage without rejecting otherwise usable products.
Attaches diagnostics to dependency-resolution results and emits them through logging.

These checks intentionally remain advisory because some constellations may have incomplete bias support while the remaining systems can still produce a useful solution.

Frequency handling

Changed the default combinations to:
- GPS G12
- GLONASS R12
- Galileo E17
- BeiDou C27
- QZSS J12
This selects Galileo E1/E5b and BeiDou B1/B2.
Fixed pdp3 command construction so frequency combinations are passed as separate arguments rather than one combined string.
Added explicit mapping-function support to PrideCLIConfig.

Same-day processing

Stages resolved RINEX 3 mixed navigation under the legacy brdmDDD0.YYp filename expected by pdp3.
Prefers BKG’s rolling same-day mixed-navigation file.
If no mixed navigation is available, limits the current-day run to GPS and GLONASS so PRIDE can use its hourly navigation builder.
Clips same-day observations to the common orbit/clock coverage endpoint with a five-minute safety margin.
Uses a physical clipped RINEX because PRIDE’s explicit time-window handling is unreliable in this macOS workflow.
Explicitly selects GMF for same-day processing, avoiding dependency on unpublished VMF grids.
Historical runs preserve an explicitly requested VMF1 selection.
Makes satellite attitude optional for near-real-time bundles.
Writes Quaternions = NONE when ATTOBX is unavailable, preventing unnecessary fallback downloads and avoiding PRIDE’s nonportable macOS sed -i failure.
Added NONE as a valid satellite-attitude configuration value.

PRIDE execution robustness

Missing required products prevent pdp3 from launching, while other dates in a batch can continue.
Handles non-UTF-8 subprocess output without aborting the job.
Replays captured PRIDE stdout and stderr through Python logging.
Detects PRIDE’s internal error: messages even when pdp3.sh exits with status zero.
Validates generated KIN files before accepting them.
Rejects incomplete SPP seed files left behind by failed downstream stages.
Handles PRIDE residual timestamps rounded to 60.0000000 seconds by carrying them into the next minute correctly.
Avoids writing literal "None" product filenames into generated PRIDE configurations.

Preserved output artifacts

Successful PRIDE runs now retain:

kin_YYYYDOY_site.kin
res_YYYYDOY_site.res
config_YYYYDOY_site.config
log_YYYYDOY_site.log
cst_YYYYDOY_site.cst
stt_YYYYDOY_site.stt
run_YYYYDOY_site.log

…ependency-resolution run. Reusing a WormHole instance will no longer prevent a previously unavailable product from being retried later. - CODE product tests now reflect the HTTPS/S3 migration and CODE/ root paths instead of obsolete FTP/year-specific paths.

frigusgulo · 2026-09-04T17:09:09Z


 logger = logging.getLogger(__name__)

+_AIUB_DOWNLOAD_ROOT = "https://www.aiub.unibe.ch/download"


These should live in yaml files and go through the propery analysis center registry

frigusgulo · 2026-09-04T17:09:31Z

        full_path = pool.full_path(directory)

        def _ls(conn: "fsspec.AbstractFileSystem") -> list[str]:
+            if hostname.rstrip("/") == _AIUB_DOWNLOAD_ROOT:


same comment from one above.

frigusgulo · 2026-09-04T17:16:10Z

      - {name: TTT, value: RAP}
      - {name: PPP, value: OPS}
-    directory: {pattern: "CODE/{YYYY}/"}
+    directory: {pattern: "CODE/"}


lets double check this one

frigusgulo · 2026-09-04T17:16:19Z

      - {name: TTT, value: RAP}
      - {name: PPP, value: OPS}
-    directory: {pattern: "CODE/{YYYY}/"}
+    directory: {pattern: "CODE/"}


lets double check this one

frigusgulo · 2026-09-04T17:16:35Z

      - {name: SMP, value: 30S}
      - {name: SMP, value: 05M}
-    directory: {pattern: "CODE/{YYYY}/"}
+    directory: {pattern: "CODE/"}


lets double check this one

frigusgulo · 2026-09-04T17:16:45Z

      - {name: SMP, value: 05M}
      - {name: SMP, value: 15M}
-    directory: {pattern: "CODE/{YYYY}/"}
+    directory: {pattern: "CODE/"}


lets double check this one

frigusgulo

lets double check those CODE product directories, and and the AIUB hard coded edge-case claude artifacts.

…rences

frigusgulo · 2026-11-02T23:00:28Z

@copilot resolve the merge conflicts in this pull request

# Conflicts: # packages/gnss-product-management/src/gnss_product_management/factories/connection_pool.py # packages/gnss-product-management/src/gnss_product_management/factories/pipelines/resolve.py # packages/gnss-product-management/test/test_code_https.py Co-authored-by: frigusgulo <43794768+frigusgulo@users.noreply.github.com>

Copilot · 2026-11-02T23:01:49Z

@copilot resolve the merge conflicts in this pull request

Merged main into the branch and pushed the merge commit. There were conflicts in connection_pool.py, pipelines/resolve.py and test_code_https.py. I kept this branch's side in each. That means the AIUB listing is still read from the YAML specs rather than the hardcoded _AIUB_* constants from main. I only checked that the code compiles. The test environment isn't installed here, so I didn't run the tests.

gottlieb added 5 commits September 2, 2026 11:56

Prefer RTS products and support CODE HTTPS archive

05bb86b

Merge branch 'add-ultra-rapid-configs' into product-bundles

940ce0e

bundle products

5e68dce

fix same day

0da3712

keep more results files

f0360cd

mikegottlieb84 requested a review from frigusgulo September 4, 2026 16:27

frigusgulo reviewed Sep 4, 2026

View reviewed changes

frigusgulo requested changes Sep 4, 2026

View reviewed changes

move hardcoded url and fix CODE paths

d46c39c

frigusgulo approved these changes Sep 9, 2026

View reviewed changes

Franklyn Dunbar and others added 4 commits September 9, 2026 10:11

ruff formatting

b257380

fix ordering

b7f5e1b

get atx from header

f9439ad

Enhance ANTEX reference validation and add tests for unversioned refe…

8758b87

…rences

frigusgulo assigned Copilot Nov 2, 2026

Copilot started work on behalf of frigusgulo November 2, 2026 23:01

View session

Copilot finished work on behalf of frigusgulo November 2, 2026 23:02

Copilot AI requested a review from frigusgulo November 2, 2026 23:02

frigusgulo merged commit 93bc5d8 into main Nov 2, 2026

frigusgulo deleted the product-bundles branch November 2, 2026 23:18

Navigation Menu

Repository navigation

Product bundles - #40

Product bundles#40
frigusgulo merged 12 commits into
mainfrom
product-bundles

mikegottlieb84 commented Sep 4, 2026

Uh oh!

frigusgulo Sep 4, 2026

Uh oh!

frigusgulo Sep 4, 2026

Uh oh!

frigusgulo Sep 4, 2026

Uh oh!

frigusgulo Sep 4, 2026

Uh oh!

frigusgulo Sep 4, 2026

Uh oh!

frigusgulo Sep 4, 2026

Uh oh!

frigusgulo left a comment

Uh oh!

frigusgulo commented Nov 2, 2026

Uh oh!

Copilot AI commented Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		logger = logging.getLogger(__name__)

		_AIUB_DOWNLOAD_ROOT = "https://www.aiub.unibe.ch/download"

Conversation

mikegottlieb84 commented Sep 4, 2026

Summary

Product bundle resolution

Product precedence and sources

Download and cache behavior

PRIDE product compatibility diagnostics

Frequency handling

Same-day processing

PRIDE execution robustness

Preserved output artifacts

Uh oh!

frigusgulo Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

frigusgulo Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

frigusgulo Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

frigusgulo Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

frigusgulo Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

frigusgulo Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

frigusgulo left a comment

Choose a reason for hiding this comment

Uh oh!

frigusgulo commented Nov 2, 2026

Uh oh!

Copilot AI commented Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants