# Review of PR #2011 "delta addons" (branch `feat-ON-124-delta-addons`)

Reviewed 2026-11-11 against master 7acfdf4 (which has #1951 merged).
Diff vs master: +581/-12 in 6 files. Of that, ~270 lines in
`oggm/utils/_workflow.py` and ~315 lines of tests.

## TL;DR

- The PR adds a **side channel** for optional gridded data. An add-on is a
  parentless tar holding one `{label}.nc` plus an `addon` manifest with a
  `grid_id`. `apply_addon` merges it into `gridded_data.nc` if the gdir's grid
  hash matches. The chain (`delta`/`standalone`) is unchanged.
- **It does not do what we need for increments.** `gridded_data.nc` is still an
  opaque file in the delta chain: any level that touches one variable ships the
  whole file again. That covers L2 with shop data, and L2 centerlines
  rewriting `glacier_mask`. Add-ons and level deltas use two unrelated
  mechanisms, with no shared code.
- **`prepro_levels` is untouched.** Nothing produces add-ons, nothing consumes
  them, and `init_glacier_directories`/`workflow` have no hook for them.
- There is one real bug in `write_group(mode="a")`: it silently replaces the
  coordinates of the whole file (confirmed below). The PR also adds two hard
  dependencies (`h5py`, `h5netcdf`) that aren't needed, plus new public
  `GlacierDirectory` API (`open_group`/`write_group`/`delete_group`) that's
  half-designed. Their docstrings still mention a "v2 format" from another
  branch.
- The useful pieces are `grid_id`, the `addon` manifest kind, the manifest-id
  pinning test, and the "add-on manifests are skipped when layering" logic.
  These are small.

**Recommendation:** fix it ourselves rather than send it back. The salvageable
part is small, and the real work is a redesign: variable-level netCDF deltas
with add-ons as a special case of them, then `prepro_levels`. That design is
ours to decide anyway. Nico could take the implementation once the design
below is agreed, but iterating on this PR as it stands won't converge.

## What the PR does

| Piece | Where | Notes |
|---|---|---|
| `GlacierDirectory.open_group / write_group / delete_group` | [_workflow.py:4235-4340](../oggm/utils/_workflow.py#L4235) | xarray + `engine="h5netcdf"`; `write_group(mode="a")` = read whole file, merge, rewrite via tmp + `os.replace` |
| `grid_id(grid)` | [_workflow.py:5781](../oggm/utils/_workflow.py#L5781) | sha256 of `json.dumps(grid.to_dict())` (proj string, x0y0, nxny, dxdy, pixel_ref) |
| manifest kind `addon`, schema 1.0 → 1.1 | [_workflow.py:5747](../oggm/utils/_workflow.py#L5747) | `grid_id` in manifest and in `manifest_id` (add-ons only, existing ids unchanged: pinned by a test) |
| `_finalize_merged_dir` skips add-on manifests | [_workflow.py:5850](../oggm/utils/_workflow.py#L5850) | so an add-on manifest is never taken as the chain leaf |
| `write_addon(gdir, label, variables, output_dir)` | [_workflow.py:6066](../oggm/utils/_workflow.py#L6066) | entity task; writes `{label}.nc` (selected `gridded_data` vars) + manifest, tars per glacier; user then calls `base_dir_to_tar` |
| `apply_addon(gdir, base_url)` | [_workflow.py:6113](../oggm/utils/_workflow.py#L6113) | downloads the 100-glacier bundle, checks `(kind, rgi_id, grid_id)`, `write_group(mode="a")`, copies manifest into gdir |
| deps `h5py`, `h5netcdf` | `pyproject.toml` | hard dependencies |
| tests | `test_deltas.py::TestAddons`, new `test_gdir.py` | |

## Does it cover increments (L2 not shipping the DEM again)?

No. On master the chain diffs whole files ([snapshot_gdir_state](../oggm/utils/_workflow.py#L5896)
hashes file bytes), so:

- L1 ships `dem.tif` + `gridded_data.nc` (topo, topo_smoothed,
  topo_valid_mask, glacier_mask, glacier_ext). #1951 moved
  `simple_glacier_masks` into L1 for this reason
  ([prepro_levels.py:1098-1103](../oggm/cli/prepro_levels.py#L1098)).
- L2 with `--add-consensus-thickness/--add-itslive-velocity/--add-millan-*/--add-hugonnet-dhdt`
  appends variables to `gridded_data.nc` → file hash changes → the **whole
  file, topo included, ships again in L2**, once per L2 flavour.
- L2 centerlines run `glacier_masks` ([prepro_levels.py:1166](../oggm/cli/prepro_levels.py#L1166)),
  which rewrites the file → same thing.
- Even a no-op rewrite changes the bytes (see finding 2: re-encoding adds
  `_FillValue` attrs), so file hashes are a poor change detector for netCDF.

The add-on code can't be reused for this as is: it has no parent link, no
removal semantics, and it's applied outside `_finalize_merged_dir`.

## Findings

Severity: **B** = bug / blocker, **D** = design issue, **M** = minor / cleanup.

### Bugs

1. **B — `write_group(mode="a")` replaces the coordinates of the whole file.**
   [_workflow.py:4307](../oggm/utils/_workflow.py#L4307) drops
   `[v for v in ds.variables if v in old.variables]` from the old dataset.
   `ds.variables` includes the coordinates `x`/`y`, so the old coordinates are
   dropped and the incoming ones win, for every variable. Probed on a copy of
   the HEF `gridded_data.nc` (netCDF4 engine swapped in, since h5netcdf isn't
   installed here): appending a variable with `x + 0.5` gave no error and no
   warning, and afterwards `x[0]` went from -4517.129 to -4516.629 for the
   whole file. `apply_addon` is protected by the `grid_id` check, but
   `write_group` is public API. Fix: drop only `ds.data_vars`, and require
   identical coordinates (`xr.merge(..., join="exact")` or an explicit
   `np.array_equal` check).

2. **M/B — mixing `decode_cf=False` (old) with decoded (new) data in the merge.**
   It works for HEF `gridded_data`: values and dtypes were identical. But
   every float variable without a `_FillValue` gains `_FillValue=nan`, and all
   variables get recompressed, so the file changes for variables nobody
   touched. The docstring advertises this for "climate and GCM data" too,
   where a raw `time` merged with a decoded `time` would break. Either read
   both decoded, or restrict the helper to gridded data.

3. **B — tests can't pass in our env, and the PR adds hard deps for no
   reason.** `h5netcdf` isn't in `oggm_env`: all 6 `test_gdir.py` tests fail
   and the 3 `TestAddons` apply tests error ("unrecognized engine
   'h5netcdf'"). The other 45 tests in the two files pass. CI would install
   it through `pip install -e .`. But `netcdf4` is already a core dependency,
   all of OGGM writes `gridded_data.nc` through netCDF4/`ncDataset`, and
   mixing two HDF5 writers on the same file is a needless risk. Drop
   `engine="h5netcdf"` and the two deps.

4. **B (latent) — `apply_addon` task name is not label-specific.** The
   entity_task logs `apply_addon` as SUCCESS, so with
   `cfg.PARAMS['auto_skip_task']=True` the *second* add-on applied to a gdir
   is silently skipped. Make the label part of the task name (e.g. pass it as
   `filesuffix`-like suffix, or don't make it an entity task with skip logic).

### Design

5. **D — add-ons and level deltas are two mechanisms.** The add-on format
   (a partial `.nc` merged into `gridded_data.nc`) is exactly what a level
   delta needs for `gridded_data`. It should be **one** mechanism: a netCDF
   *patch* whose precondition is either a parent id (level delta) or a
   `grid_id` (add-on). See the proposal below.

6. **D — `grid_id` is computed at apply time, never recorded in the chain.**
   Neither L1 nor any other chain manifest stores its `grid_id`, so you can't
   tell whether an add-on fits without downloading the gdir. Issue #2017
   (catalogue) needs it in the tree index too. Store `grid_id` in every chain
   manifest, or at least in the root (L1). It is set at L1 by
   `define_glacier_region` and never changes. Caveats:
   - The hash includes the proj string and raw floats
     (`x0y0 = (-4517.129122282515, 5187040.526772606)` for HEF). Identical
     inputs give identical floats, but a different PROJ/pyproj that formats
     `+lon_0=10.7584` differently changes the id. It's documented, and
     acceptable as long as add-ons are built from the published L1 rather
     than recomputed.
   - The grid depends on outlines, border and dx rules, not on the DEM. So
     one add-on tree per (RGI version, border) serves every DEM flavour. That
     is worth stating explicitly, and it's the "grid family" question of #2017.

7. **D — provenance.** The add-on manifest records the label and file hash
   only: no variable list, no source product/version, no `fetched_from`
   (which #1951 added for chain manifests). An L2 that *binned* these
   variables into `elevation_band_flowline.csv` doesn't record which add-on
   (id) it used. That matters because applying an add-on later **does not
   re-bin**: the binned values stay those of L2's build. Both should be
   recorded.

8. **D — no client integration.** Users must know the add-on tree URL and
   call `apply_addon` per gdir. We need either
   `init_glacier_directories(..., addons=[...])` or a
   `workflow.apply_addons(gdirs, [...])` with default URLs, plus a documented
   tree layout (e.g. `addons/<product>/<version>/RGI62/b_080/RGI60-11/RGI60-11.008.tar`).
   The PR leaves the layout to the caller.

9. **D — new public API on `GlacierDirectory` is half-designed.**
   `open_group`/`write_group`/`delete_group` are used only by the add-on code.
   The docstrings say "Preferred way to read gridded, climate and GCM data"
   and contain a `TODO: ... maybe the method name should be something
   clearer` and a `ponytail:` comment. `test_gdir.py` is titled "Tests for the
   v2 glacier directory format" (leftover from #1966?). Nico asked us about
   the naming (`open_dataset`, `gdir.datasets[...]`). I'd keep these
   **private** helpers here (`_merge_nc`) and decide the public API with the
   v2 format discussion, not in this PR.

10. **D — interaction with later levels is undefined.** If a user applies an
    add-on and then builds a level on that gdir, the add-on variables end up
    in that level's `gridded_data.nc`. With today's whole-file deltas they
    ship inside the delta, but the add-on manifest doesn't (deltas only
    include their own manifest). A standalone (L5) ships the add-on manifest
    but maybe not its data. A level layered on top that updates
    `gridded_data.nc` wipes the add-on variables but leaves the add-on
    manifest behind. Variable-level patches fix most of this naturally.

### Minor

11. `write_addon` takes `border`/`rgi_version` from `cfg.PARAMS`, not from the
    gdir, so it can mislabel when building from a gdir with another border.
    Same pattern as the chain, but the gdir knows its own values.
12. `apply_addon`: `(mf,) = glob(...)` gives an opaque `ValueError` for 0 or 2
    manifests. It doesn't verify the shipped `.nc` hash against the manifest
    (cheap, and the chain doesn't either). `tar_base[:-4]` assumes `.tar`.
13. `open_group` sets `chunks=None` by default, which is already xarray's
    default: the "dask fancy indexing" comment is moot.
14. `get_store_filepath` `filesuffix or ''`: harmless unrelated fix, fine.
15. Schema bump 1.0 → 1.1 is semantically right (old readers already raise
    on the unknown kind). If nothing has been published with 1.0 yet, we
    could leave it at 1.0. Check whether the "convert one region" trial
    from #1951 was published.
16. Tests: no test for a coordinate mismatch (finding 1), two add-ons on one
    gdir, `auto_skip_task`, an add-on applied on top of a real layered chain
    (`init_glacier_directories(from_tar=...)` + `apply_addon`), or an add-on
    variable clashing with an existing one. `test_gdir.py` sets
    `cfg.PATHS["working_dir"]` for nothing. The HEF copies in `TestAddons`
    are done right (`tmp_path` copies, the shared fixture isn't touched).

## Proposal: variable-level netCDF deltas, add-ons as a special case

1. **Per-variable state for netCDF groups.** In `snapshot_gdir_state`, for
   `gridded_data.nc` (a small allow-list), emit one key per variable, e.g.
   `gridded_data.nc#topo`, hashed from *decoded content* (dims, shape, dtype,
   data bytes, attrs), plus one key for coords + global attrs
   (`gridded_data.nc#@coords`). This is robust to engine, compression and
   `_FillValue` noise. As a bonus, identical rebuilds get identical ids.
2. **Patch files in the tar.** A delta whose `gridded_data` entries changed
   ships `gridded_data.<label>.patch.nc` holding only the added/updated
   variables, and lists removed variables in the manifest. It can't ship them
   as `gridded_data.nc`, because `_extract_tars` layers tars by overwriting
   files.
3. **Apply in `_finalize_merged_dir`.** After the chain is validated, merge the
   patches root → leaf into `gridded_data.nc` (exact coords required), drop
   removed variables, delete the patch files. Readers are unchanged: after
   layering, `gridded_data.nc` is a normal file.
4. **Add-on = patch + `grid_id` precondition.** Same file format, same merge
   function. `apply_addon` = fetch, check `grid_id`, merge, record the
   manifest (with `fetched_from`). Store `grid_id` in L1 (and children) so
   the check and the catalogue (#2017) need no download.
5. **Effect:** L2 elev-bands with no shop products ships no gridded data; L2
   centerlines ships `glacier_mask`/`glacier_ext` (+ whatever `glacier_masks`
   really changes), not topo; L1 stays the only carrier of the DEM grid.

Open question: also split `dem.tif` vs `gridded_data#topo` duplication (both
in L1)? Out of scope here, but worth a note.

## What `prepro_levels` needs

Today L2 runs the shop tasks, bins their variables into the flowline CSVs,
and ships them inside `gridded_data.nc`
([prepro_levels.py:1168-1206](../oggm/cli/prepro_levels.py#L1168)). The shop
variables only depend on the grid (they use `glacier_mask` only for the
summary statistics), so `grid_id` is the right anchor for them.

Option A (minimal): keep the shop tasks in L2. After binning, write one add-on
per product (`write_addon` → `addons/<product>/...`), then **drop those
variables from `gridded_data.nc` before the L2 manifest**, so the published L2
state equals the builder's state. L2's manifest records the add-on ids it
binned.

Option B (preferred, I think): a separate add-on stage built **from L1**
(only the grid is needed, plus masks for the statistics), e.g.
`run_prepro_addons(products=[...])` writing `addons/<product>/<version>/RGI62/b_080/`
and the `*_statistics_{reg}.csv` summaries. L2 flavours then
`apply_addon` what they want to bin and strip it again before writing (as in
A). A new product (or a new version of ITS_LIVE) then never needs an L2
rebuild, and one add-on tree serves all L2 flavours on that L1.

Either way:
- the `add_*` CLI flags become "bin these add-ons" at L2 plus "build these
  add-ons" in the add-on stage;
- `add_glathida` (point CSV) and `rgi7g_to_complex` aren't gridded and don't
  fit the `.nc` add-on. Keep them in L2 for now, or design a file-level
  add-on later;
- the summary statistics move with the add-on stage.

## Decisions for us

1. One mechanism (patches with a parent-or-grid precondition) vs keeping
   add-ons separate. I recommend one.
2. Public group API now, or private helpers until the v2 format? I recommend
   private.
3. `prepro_levels` option A or B. I lean B.
4. Add-on tree layout + default URLs + client hook (`init_glacier_directories(addons=...)`?).
5. `grid_id` in chain manifests (needed for #2017 anyway).
6. Who implements: us (probably) or Nico against this design.

## Checks run

- `.cluster_env/run.sh python -m pytest -p no:cacheprovider -q oggm/tests/test_gdir.py oggm/tests/test_deltas.py`:
  45 passed, 6 failed, 3 errors. All failures and errors are the missing
  `h5netcdf` engine.
- Probe script (scratch, netCDF4 engine swapped in) on a copy of the HEF
  `gridded_data.nc` from the test cache: confirmed finding 1 (coordinate
  replacement) and finding 2 (`_FillValue` attrs added, values and dtypes
  intact).
- Not run: anything with real prepro data, `prepro_levels` tests, slow tests.

## Status (2026-11-11, after the review)

Implemented on `feat-ON-124-delta-addons` (local commits, not pushed),
following the plan agreed with Fabien (add-ons on L1-L4, refused on L5;
flowline geometry independent of binned data; binned values in the existing
csv files; shop products moved out of L2 into an add-on stage):

1. `afcea2a6` netCDF patch helpers replace the group API; h5py/h5netcdf
   dropped; coordinate check (fixes finding 1).
2. `88552536` `tasks.bin_gridded_variables` + decoupled
   `elevation_band_flowline` geometry.
3. `caaa2eec` per-variable `gridded_data` deltas in the chain
   (`gridded_data.{label}.patch.nc`), `grid_id` in every manifest, add-ons
   rebuilt on patches (`write_addon`, `apply_addon`, `workflow.apply_addons`,
   `utils.ADDON_URLS`), `entity_task(name_kwarg=...)`.
4. `2defd16a` `oggm_prepro_addons` (`oggm/cli/prepro_addons.py`), shop flags
   removed from `oggm_prepro`.
5. `1a106433` docs.

Open: `ADDON_URLS` is empty until add-on trees are published; schema kept at
1.1 (assumes no delta tree published with 1.0).

Done since: `7e847ccf` `preserve_totals` is set per variable in the add-on
manifest (False for the ITS_LIVE and Millan velocities, True otherwise).
`5a3df7cf` merge of upstream/master. `2cc816a9` file add-ons (GlaThiDa moved
out of L2 into `oggm_prepro_addons`) and per-region add-on indexes
(`RGI{v}/b_{border}/index/{region}.csv`, written by `bundle_addon`).
