Future Direction for CDI-pio - developer meeting 20120323¶

The following aspects were discussed:

• For CDO the capability to open
  many files simultaneously is required to merge partitioned output,
  it's not yet clear how to test this.
• The example programs should be extended into test cases. For
  robustness, data should be compared after a write/read cycle per some
  hashing function. (DKRZ, #2153)
• Some files need to be prepared and be part of the source
  distribution to test read functionality independently. (DKRZ, #2163)
• An example for multi-thread use of the library will be
  prepared. (Uwe Schulzweida, #2164)
• Appending data to a file is not covered by the examples, this will
  need to be tested additionally.
• It would be beneficial to have the parallel output ready for the
  on-going STORM
  experiments. But since remaining experiments require the coupled
  model, this seems ambitiuous.
• Serial tests will provide a reference for the result of parallel tests.
• Since the different output formats use essentially the same
  interface, the tests will be written such that one can run each test
  for every file format.
  • This includes 7 variants of writing Grib1, of which only the MPI
    non-blocking variant will be tested at first.
  • For netCDF only the long offsets format and netCDF 4 can be tested
    in the parallel variant. Which can only be compiled in an either/or
    fashion with respect to the parallel netCDF variants.
  • Grib2 support should be added.
• Tests should be prepared for different layers of the parallel
  I/O. These layers are:
  1. Transfer of decomposed data on the compute processes to the I/O
     server processes.
  2. file format dependent de-/encoding of data
  3. performing the actual write within the
     POSIX/MPI-IO/Filesystem-specific fashion

• DKRZ participants will give feedback for the documentation of
  CDI-pio on next meeting.