Skip to content

Multi-channel select_mean via deferred Derived put_db - #82

Open
jjguerrette wants to merge 1 commit into
JCSDA:developfrom
jjguerrette:feature/select_mean_multichannel
Open

jjguerrette wants to merge 1 commit into
JCSDA:developfrom
jjguerrette:feature/select_mean_multichannel

Conversation

@jjguerrette

Copy link
Copy Markdown

Description

Extends Gaussian Thinning select_mean so a single channelled filter variable (YAML Variable with a channel list, filtervars.size() == 1, filtervars.nvars() >= 1) gets a per-channel arithmetic mean in DerivedObsValue, with one shared keep-mask for MetaData.

Previously select_mean only used filtervars.variable(0). Writing DerivedObsValue channel-by-channel during the gather loop is unsafe: IODA prefers Derived over ObsValue, and ObsAccessor (used by Gaussian Thinning) does not pass skipDerived=true, so later channel reads see a partial Derived product instead of the original ObsValue.

This PR changes only Gaussian_Thinning.cc (parameters API unchanged):

  1. Compute phase — for each expanded component jv = 0 … nvars()-1, gather ObsValue (and optional error fields), run identifyThinnedObservationsMean, and build local mean/error vectors without put_db.
  2. Shared keep-mask — use the keep-mask from jv == 0 for thinning flags (one geometry/priority decision per cell).
  3. Write phase — after all components are computed, put_db all DerivedObsValue / optional DerivedObsError.
  4. Error-variable contract — if uncertainty is enabled, require random/systematic error variables to match the filter variable’s size and channel list.
  5. Indexing — initialize DerivedObsError to missing for all nlocs and write by localObsId (hardens sparse apply layouts).

Scalar / single-component select_mean (nvars() == 1) is unchanged in contract. Adds MPI ctest qc_gauss_thinning_mean_multichannel.

Out of scope / known orthogonal issue: During development, I found qc_gauss_thinning_mean_error EffectiveError layout mismatches under float geodesic distance ties (GeodesicDistanceCalculator + std::min_element first-of-ties). That failure reproduces on my local build without this PR diff and is not addressed here. There are several possible reasons for that failure and possible solutions, in case it arises during review of this PR.

Issue(s) addressed

N/A (no linked issue)

Dependencies

  • None

Impact

Expected impact on downstream repositories:

  • Enables analysis-time multi-channel select_mean for radiance (e.g. brightness temperature with a channel list) in observer YAML.
  • No YAML parameter changes; existing scalar select_mean configs should behave as before.

Checklist

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have run the unit tests before creating the PR

Loop filtervars.nvars() for cell means, share jv==0 keep-mask, and put_db
all DerivedObsValue/DerivedObsError after the compute phase so IODA Derived
preference cannot poison later channel ObsValue reads. Index DerivedObsError
by nlocs. Add qc_gauss_thinning_mean_multichannel ctest.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jjguerrette

Copy link
Copy Markdown
Author

@mo-RowanMcL @BenjaminRuston @twsearle GitHub suggests you contributed to this segment of Gaussian_Thinning.cc recently. So perhaps you can review or suggest others.

A little more info may be useful. Currently I am attempting to combine two Gaussian Thinning steps, the first doing super-obbing by select_mean, and the second doing simple thinning:

      - filter: Gaussian Thinning
        filter variables:
        - name: brightnessTemperature
          channels: 1-12
        horizontal_mesh: 50.0
        min_num_obs_per_bin: 1
        select_mean: true
      - filter: Gaussian Thinning
        horizontal_mesh: 121
        use_reduced_horizontal_grid: true
        distance_norm: geodesic
        action:
          name: reduce obs space

Without this PR, the Variational application provides the following log output:

Jo Observations:
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_1: Nobs=     5794, Min= 1.766797e+02, Max= 2.784375e+02, RMS= 2.377325e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_2: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_3: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_4: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_5: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_6: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_7: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_8: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_9: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_10: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_11: No observations.
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_12: No observations.
...
CostJo Observations: 
CostJo   : Nonlinear Jo(tomorrow-tms_TMS-S02) = 2399.96, nobs = 5794, Jo/n = 0.414215, err = 10.3824
CostJo   : Nonlinear Jo = 2399.96
CostJo   : Nonlinear Jo(tomorrow-tms_TMS-S02) = 1334.115319, nobs = 5794, Jo/n = 0.230258, err = 10.382372
CostJo   : Nonlinear Jo = 1334.115319
CostJo   : Nonlinear Jo(tomorrow-tms_TMS-S02) = 1341.851753, nobs = 5794, Jo/n = 0.231593, err = 10.382372
CostJo   : Nonlinear Jo = 1341.851753

The hard-coded single-variable implementation yields missing values for DerivedObsValue of channels after the first. I tried chaining multiple select_mean filters together, one for each channel, but the ObsAccessor -> ObsSpace API locks Gaussian_Thinning.cc to retrieve missing DerivedObsValue vectors for the subsequent channels.

With this PR, the output changes to:

Jo Observations:
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_1: Nobs=     5823, Min= 1.781406e+02, Max= 2.780039e+02, RMS= 2.377027e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_2: Nobs=     5452, Min= 2.203047e+02, Max= 2.762031e+02, RMS= 2.602705e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_3: Nobs=     5426, Min= 2.297969e+02, Max= 2.729351e+02, RMS= 2.603439e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_4: Nobs=     5552, Min= 2.298092e+02, Max= 2.667799e+02, RMS= 2.543354e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_5: Nobs=     5884, Min= 2.221172e+02, Max= 2.536719e+02, RMS= 2.414510e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_6: Nobs=    10398, Min= 2.039844e+02, Max= 2.364327e+02, RMS= 2.240597e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_7: Nobs=    10440, Min= 2.036283e+02, Max= 2.335807e+02, RMS= 2.146909e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_8: Nobs=    10429, Min= 2.007402e+02, Max= 2.387552e+02, RMS= 2.203731e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_9: Nobs=    10443, Min= 2.073932e+02, Max= 2.652897e+02, RMS= 2.423684e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_10: Nobs=     5950, Min= 2.314401e+02, Max= 2.798919e+02, RMS= 2.584670e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_11: Nobs=     5875, Min= 2.401113e+02, Max= 2.871609e+02, RMS= 2.689595e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_12: Nobs=     5718, Min= 2.349007e+02, Max= 2.933574e+02, RMS= 2.751740e+02
...
CostJo Observations: 
CostJo   : Nonlinear Jo(tomorrow-tms_TMS-S02) = 25385.4, nobs = 87390, Jo/n = 0.290484, err = 3.65712
CostJo   : Nonlinear Jo = 25385.4
CostJo   : Nonlinear Jo(tomorrow-tms_TMS-S02) = 16011.580971, nobs = 87390, Jo/n = 0.183220, err = 3.657116
CostJo   : Nonlinear Jo = 16011.580971
CostJo   : Nonlinear Jo(tomorrow-tms_TMS-S02) = 14808.914324, nobs = 87390, Jo/n = 0.169458, err = 3.657116
CostJo   : Nonlinear Jo = 14808.914324

When I remove the select_mean filter entirely (develop, or this PR):

Jo Observations:
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_1: Nobs=     5847, Min= 1.789922e+02, Max= 2.784375e+02, RMS= 2.376721e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_2: Nobs=     5448, Min= 2.277969e+02, Max= 2.760547e+02, RMS= 2.603554e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_3: Nobs=     5389, Min= 2.357969e+02, Max= 2.736250e+02, RMS= 2.603834e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_4: Nobs=     5549, Min= 2.300547e+02, Max= 2.673281e+02, RMS= 2.543187e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_5: Nobs=     5900, Min= 2.184844e+02, Max= 2.540469e+02, RMS= 2.414134e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_6: Nobs=    10456, Min= 2.041953e+02, Max= 2.366875e+02, RMS= 2.240314e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_7: Nobs=    10484, Min= 2.028750e+02, Max= 2.344531e+02, RMS= 2.146619e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_8: Nobs=    10429, Min= 1.996484e+02, Max= 2.406406e+02, RMS= 2.203586e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_9: Nobs=    10503, Min= 2.057031e+02, Max= 2.653750e+02, RMS= 2.423799e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_10: Nobs=     5969, Min= 2.311094e+02, Max= 2.808828e+02, RMS= 2.585280e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_11: Nobs=     5902, Min= 2.386797e+02, Max= 2.875703e+02, RMS= 2.690512e+02
Jo Obs :tomorrow-tms_TMS-S02:brightnessTemperature_12: Nobs=     5727, Min= 2.316875e+02, Max= 2.933516e+02, RMS= 2.753434e+02
...
CostJo Observations: 
CostJo   : Nonlinear Jo(tomorrow-tms_TMS-S02) = 33372.7, nobs = 87603, Jo/n = 0.380954, err = 3.65196
CostJo   : Nonlinear Jo = 33372.7
CostJo   : Nonlinear Jo(tomorrow-tms_TMS-S02) = 25282.556047, nobs = 87603, Jo/n = 0.288604, err = 3.651962
CostJo   : Nonlinear Jo = 25282.556047
CostJo   : Nonlinear Jo(tomorrow-tms_TMS-S02) = 23977.423039, nobs = 87603, Jo/n = 0.273706, err = 3.651962
CostJo   : Nonlinear Jo = 23977.423039

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant