CDO

Hi Valeriu!

HFD5-version mismatch is a common behavior - unfortunately. The reason is, that the header files used at compile time do not fit to the version used at run-time thought dynamic linking. Since I don't know about your building and running environment I cannot say more at the moment

what it shown in the output of cdo -V if evaluated at runtime though H5get_libversion()

hth
ralf

Hi Ralf,

Cheers muchly for your reply! Our environment is fairly complex but it all boils down to a conflict between cdo=1.9.7.1 compiled with hdf5=1.10.5 and mpi with mpich rather than openmpi - take for instance this minimal conda environment file:

---
name: cdoenv
channels:
  - conda-forge

dependencies:
  - hdf5=1.10.5
  - cdo=1.9.7.1
  - mpich

This environment can not be solved because the new version of cdo needs mpi with openmpi rather than mpich; if installing cdo=1.9.6 that will work but, as I said above, the hdf5=1.10.5 in the environment will make cdo throw the version mismatch warning and make it stop. Any advice on what to do to use the latest cdo, latest hdf5 and mpi with mpich would be greatly appreciated. Many thanks in advance!

Cheers,
Val

CDO does not need MPI at all - don't know why it is included. I guess, because other users of hdf5 need it. Do you need MPI?

BTW: I see multiple hdf5 build on conda-forge:

hdf5                          1.10.5 mpi_mpich_ha7d0aea_1000  conda-forge         
hdf5                          1.10.5 mpi_openmpi_hac320be_1000  conda-forge         
hdf5                          1.10.5 nompi_h3c11f04_1100  conda-forge     

I cannot create the env at all

% cat cdoenv.txt 
---
name: cdoenv
channels:
  - conda-forge
dependencies:
  - hdf5=1.10.5
  - cdo=1.9.7.1
(base) <ram@melian:~/src/tmp>
% conda env create --file cdoenv.txt 

SpecNotFound: Environment with requirements.txt file needs a name

well here's what conda says when I try creating the environment above:

$ conda env create -n cdoenv -f cdoenv.yml 
Collecting package metadata (repodata.json): done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package mpi conflicts for:
mpich -> mpi==1.0=mpich
hdf5=1.10.5 -> openmpi[version='>=3.1,<3.2.0a0'] -> mpi==1.0=openmpi
cdo=1.9.7.1 -> eccodes -> libnetcdf[version='>=4.6.2,<4.6.3.0a0,>=4.6.2,<4.7.0a0'] -> hdf5[version='>=1.10.4,<1.10.5.0a0,>=1.10.5,<1.10.6.0a0'] -> openmpi[version='>=3.1,<3.2.0a0'] -> mpi==1.0=openmpi

We need mpi with mpich in our tool's environment that can only be installed only if I relax the cdo version to no version specified:

(cdoenv) valeriu@valeriu-PORTEGE-Z30-C:~$ conda list mpi
# packages in environment at /home/valeriu/anaconda3/envs/cdoenv:
#
# Name                    Version                   Build  Channel
mpi                       1.0                       mpich    conda-forge
mpich                     3.2.1             hc99cbb1_1013    conda-forge
(cdoenv) valeriu@valeriu-PORTEGE-Z30-C:~$ conda list cdo
# packages in environment at /home/valeriu/anaconda3/envs/cdoenv:
#
# Name                    Version                   Build  Channel
cdo                       1.9.6             ha44cbb3_1008    conda-forge
(cdoenv) valeriu@valeriu-PORTEGE-Z30-C:~$ conda list hdf5
# packages in environment at /home/valeriu/anaconda3/envs/cdoenv:
#
# Name                    Version                   Build  Channel
hdf5                      1.10.5          nompi_h3c11f04_1100    conda-forge

it's gotta be a yml file not a text, you can create from txt by invoking conda create rather than conda env create but in that case the txt file should contain pointers to the html locations of packages, yml is more convenient

Here is the most barebones environment that needs the latest cdo and mpich:

cdoenv.yml ->

---
name: cdoenv
channels:
  - conda-forge

dependencies:
  - cdo=1.9.7.1
  - mpich

That results in solving failure:

<prUnsatisfiableError: The following specifications were found to be incompatible with each other:

Package mpi conflicts for:
mpich -> mpi==1.0=mpich
cdo=1.9.7.1 -> eccodes -> libnetcdf[version='>=4.6.2,<4.6.3.0a0,>=4.6.2,<4.7.0a0'] -> hdf5[version='>=1.10.4,<1.10.5.0a0,>=1.10.5,<1.10.6.0a0'] -> openmpi[version='>=3.1,<3.2.0a0'] -> mpi==1.0=openmpie>

So it looks to me like you guys built the latest cdo usinc eccodes that are built hdf5_mpi_openmpi rather than nompi

My apologies, it is the fftw that is the problem - by installing cdo=1.9.7.1 the fftw built with openmpi comes in the box and that's the bit that doesn't allow for installs with mpich, here's an environment in which I installed only cdo=1.9.7.1:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
bzip2                     1.0.8                h516909a_0    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
cdo                       1.9.7.1              ha44cbb3_0    conda-forge
curl                      7.65.3               hf8cf82a_0    conda-forge
eccodes                   2.13.1               h83cb724_0    conda-forge
expat                     2.2.5             he1b5a44_1003    conda-forge
fftw                      3.3.8           mpi_openmpi_h6dd7431_1006    conda-forge
hdf4                      4.2.13            h9a582f1_1002    conda-forge
hdf5                      1.10.5          nompi_h3c11f04_1100    conda-forge
icu                       64.2                 he1b5a44_0    conda-forge
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
krb5                      1.16.3            h05b26f9_1001    conda-forge
libaec                    1.0.4                hf484d3e_0    conda-forge
libcurl                   7.65.3               hda55be3_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
libiconv                  1.15              h516909a_1005    conda-forge
libnetcdf                 4.6.2             h056eaf5_1002    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
libxml2                   2.9.9                hee79883_2    conda-forge
mpi                       1.0                     openmpi    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
openmpi                   4.0.1                hc99cbb1_1    conda-forge
openssl                   1.1.1c               h516909a_0    conda-forge
ossuuid                   1.6.2             hf484d3e_1000    conda-forge
proj4                     6.1.0                he751ad9_2    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
sqlite                    3.29.0               hcee41ef_0    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
udunits2                  2.2.27.6          h4e0c4b3_1001    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h516909a_1005    conda-forge

and here is an environment in which I installed mpi with mpich, fftw and eccodes to reflect the dependencies of cdo (but not installing cdo per se):

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
bzip2                     1.0.8                h516909a_0    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
curl                      7.65.3               hf8cf82a_0    conda-forge
eccodes                   2.13.1               h83cb724_0    conda-forge
fftw                      3.3.8           nompi_h7f3a6c3_1106    conda-forge
hdf4                      4.2.13            h9a582f1_1002    conda-forge
hdf5                      1.10.5          nompi_h3c11f04_1100    conda-forge
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
krb5                      1.16.3            h05b26f9_1001    conda-forge
libaec                    1.0.4                hf484d3e_0    conda-forge
libcurl                   7.65.3               hda55be3_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
libnetcdf                 4.6.2             h056eaf5_1002    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
mpi                       1.0                       mpich    conda-forge
mpich                     3.2.1             hc99cbb1_1013    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
openssl                   1.1.1c               h516909a_0    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
zlib                      1.2.11            h516909a_1005    conda-forge

Would it be possible to alter the cdo dependency requirements to allow for fftw built with nompi and/or mpich? Many thanka again and my apoligies for all the hassle and multiple messages :)
Cheers,
Val

in the conda package in info/index.json:

{
  "arch": "x86_64",
  "build": "ha44cbb3_0",
  "build_number": 0,
  "depends": [
    "eccodes",
    "fftw * mpi_openmpi_*",
    "jasper >=1.900.1,<2.0a0",
    "libgcc-ng >=7.3.0",
    "libgfortran-ng >=7,<8.0a0",
    "libnetcdf >=4.6.2,<4.6.3.0a0",
    "libstdcxx-ng >=7.3.0",
    "libxml2 >=2.9.9,<2.10.0a0",
    "ossuuid",
    "proj4 >=6.1.0,<6.1.1.0a0",
    "udunits2" 
  ],
  "license": "GPL-2.0",
  "name": "cdo",
  "platform": "linux",
  "subdir": "linux-64",
  "timestamp": 1561556830831,
  "version": "1.9.7.1" 
}

Well - that ends up being pretty conda-specific. I maintain the conda-forge build of cdo, but never touched the versioning of the dependencies. Let's ask the other maintainer, he is more experienced with this.

Is This One from you?

Indeed, a conda conundrum
Very much appreciated, guys! I reckon that if you allow the fftw build against a non-specific mpi (like it was for 1.9.6) it shouldn't affect the cdo installation and functionality (I may be very very wrong though haha), but limiting the package builds to specific mpi's will make users like us be unable to construct the environment anymore...
Cheers again! V

Please comment on the github issue if possible. this is valuable information for conda-forge!!

No, that gitHub issue is not mine, I revised mine here: https://github.com/ESMValGroup/ESMValTool/issues/1239
But it seems that person is hitting the same wall (albeit I doubt it's `libnetcdf` that's the problem but rather the mpi_openmpi set for fftw for us)

I have also commented on that particular gitHub issue you pointed me to, I can conform that a barebones conda environment with only hdf5=1.10.5 and cdo=1.9.7.1 as deps solves and build fine for me on a Linux machine, so that one seems an OSX-specific issue

thx - we should go for the source of the problem and drop MPI. I convinced the debian maintainers to do this long before.

Huzzah! will be waiting for your new conda package and buy you a beer if all works fine

challenge accepted B-)