Project

General

Profile

Future Direction for CDI-pio - developer meeting 20120323

The following aspects were discussed:

  • For CDO the capability to open
    many files simultaneously is required to merge partitioned output,
    it's not yet clear how to test this.
  • The example programs should be extended into test cases. For
    robustness, data should be compared after a write/read cycle per some
    hashing function. (DKRZ, #2153)
  • Some files need to be prepared and be part of the source
    distribution to test read functionality independently. (DKRZ, #2163)
  • An example for multi-thread use of the library will be
    prepared. (Uwe Schulzweida, #2164)
  • Appending data to a file is not covered by the examples, this will
    need to be tested additionally.
  • It would be beneficial to have the parallel output ready for the
    on-going STORM
    experiments. But since remaining experiments require the coupled
    model, this seems ambitiuous.
  • Serial tests will provide a reference for the result of parallel tests.
  • Since the different output formats use essentially the same
    interface, the tests will be written such that one can run each test
    for every file format.
    • This includes 7 variants of writing Grib1, of which only the MPI
      non-blocking variant will be tested at first.
    • For netCDF only the long offsets format and netCDF 4 can be tested
      in the parallel variant. Which can only be compiled in an either/or
      fashion with respect to the parallel netCDF variants.
    • Grib2 support should be added.
  • Tests should be prepared for different layers of the parallel
    I/O. These layers are:
    1. Transfer of decomposed data on the compute processes to the I/O
      server processes.
    2. file format dependent de-/encoding of data
    3. performing the actual write within the
      POSIX/MPI-IO/Filesystem-specific fashion
  • DKRZ participants will give feedback for the documentation of
    CDI-pio on next meeting.