[franzpoeschel]

We currently don't support memory selections yet, e.g. when you have a 2D memory buffer but want to write only an arbitrary block from it. That block is then non-contiguous, but ADIOS2 has so-called memory selections to support this.

We currently don't have an API that would be able to expose this (except in Python where the native Python buffer protocol is powerful enough to express this; we currently throw an error when detecting this strides in selection are inefficient, not implemented!). Since the loadChunk()/storeChunk() API is somewhat convoluted by now anyway, I didn't want to add even further overloads.

New API Design

The new API is centered around RecordComponent::prepareLoadStore(), which returns a configuration object that can be chained with various methods before executing the actual load/store operation.

Methods return an auxiliary::DeferredComputation<T> object that encapsulates the operation without immediately executing it. The computation is only performed when get() (or operator()()) is called on the returned object:

// Returns DeferredComputation<std::shared_ptr<int>>
auto result = E_y.prepareLoadStore()
    .offset({0, 5})
    .extent({1, 5})
    .load<int>();

// Execution happens here when get() is called
auto data = result.get();

Configuration Methods

The configuration chain supports:

• offset(Offset) - Set the offset within the dataset (optional)
• extent(Extent) - Set the extent within the dataset (optional)
• memorySelection(MemorySelection) - Set memory selection for non-contiguous buffers (only available after withSharedPtr(), withRawPtr(), or withContiguousContainer())
• unsafeNoAutomaticFlush(): The returned DeferredComputation object will be a no-op object. Buffers will still be returned, but they might not yet be written/read until manually flushing.

Buffer Specification Methods

If not specifying a buffer, the following allocating load/store operations are available:

• storeSpan<T>() - Returns DynamicMemoryView<T> (a span-like view)
• load<T>() - Returns DeferredComputation<std::shared_ptr<T>>
• loadVariant() - Returns DeferredComputation<variant> for type-erased loading
• load<T>(policy) - Returns std::shared_ptr<T>>

Alternatively, you may specify a buffer using one of:

• withSharedPtr(std::shared_ptr<T>) - For shared pointer managed buffers
• withUniquePtr(UniquePtrWithLambda<T>) - For unique pointer with custom deleter
• withRawPtr(T*) - For raw pointer buffers
• withContiguousContainer(Container&) - For contiguous containers (automatically deduces size if extent is not set)

The following operations are only available after specifying a buffer. Template parameters are no longer needed as the type is given through the buffer type:

• store() / load() - Returns DeferredComputation<void>

(Load operations are only available after specifying a writable buffer, i.e. raw/shared pointer or contiguous container of non-const type)

ADIOS2 Memory Selection Support

This PR adds support for ADIOS2 memory selections, allowing writing from non-contiguous memory blocks:

std::shared_ptr<int> data;
E_y.prepareLoadStore()
    .offset({0, 5})
    .extent({1, 5})
    .withSharedPtr(data)
    .memorySelection({{1, 1}, {5, 5}})  // Write block from within the buffer
    .store();

Note: Memory selection support depends on ADIOS2 version. For ADIOS2 versions that cannot reset memory selections (PR ornladios/ADIOS2#4169), a warning is displayed. The API gracefully handles both cases.

Migration from Old API

The old loadChunk() and storeChunk() methods are now fully implemented using the new API internally, providing full backward compatibility while benefiting from the new implementation. Existing code continues to work without changes.

Examples of Old to New Migration

Old:

std::shared_ptr<int[]> data;
E_y.loadChunk(data, {0, 5}, {1, 5});

New:

auto result = E_y.prepareLoadStore()
    .offset({0, 5})
    .extent({1, 5})
    .withSharedPtr(data)
    .load<int>();
// Later: result.get();

Old:

std::vector<int> data(100);
E_y.storeChunk(data, {0, 5});

New:

E_y.prepareLoadStore()
    .offset({0, 5})
    .extent({1, 5})
    .withContiguousContainer(data)
    .store();

Design Considerations & Open Questions

Memory Selection Limitations

• HDF5 backend throws an error when memory selection is specified (not supported by HDF5)
• ADIOS2 requires version with memory selection reset support for clean reuse of variables
• Memory selection only available after buffer specification (not on bare prepareLoadStore())

Files Changed

• include/openPMD/LoadStoreChunk.hpp - New header with the main API
• include/openPMD/LoadStoreChunk.tpp - Template implementations
• include/openPMD/auxiliary/Future.hpp - New DeferredComputation wrapper
• include/openPMD/Dataset.hpp - Added MemorySelection struct
• include/openPMD/RecordComponent.hpp - Integration of new API
• src/LoadStoreChunk.cpp - Non-template implementations
• src/auxiliary/Future.cpp - DeferredComputation implementation
• All IO backends (ADIOS2, HDF5, etc.) - Memory selection support

TODO

• What about Python? The buffer protocol in Python may already support this; consider keeping Python API unchanged for now
• Decide on deferred vs. immediate API style
• Consider introducing .noop_future() option to disable future return types for cases where you don't need to track completion
• Further binary size optimization

Related

• ADIOS2 PR ornladios/ADIOS2#4169: Memory selection reset support

TODO:

• look at hpx future for inspiration