CDO

hi Marco!

Brandon (author of https://code.mpimet.mpg.de/issues/10069) wanted to work on this, too. so maybe you get in contact with each other.

The dependencies look normal to me, but your configure call has some flaws IMO:

• --without-threads this is obsolete, in fact it will turn CDO into a single-thread application, which is not what people want. please, drop this options
• --enable-cdi-lib CDO comes with its on CDI code out-of-the-box. Sometimes this code is not a CDI-release on its own, so you will not be able to link every CDO release to a certain CDI release. This configure switch only exist to have the option to build CDO against an existing CDI, but it should not be taken as the default way to build CDO
• --enable-iso-c-interface this is a special CDI option. CDO does not need that switch. The background is: CDI is a libeary written in C but also offers a Fortran interface. With this switch you can tweak the way the fortran interface is build. CDO is plain C++, so CDI does not need to provide the Fortran at all.

Please adjust the options and run the tests again.

thx in advance
ralf

Hi Ralf,

except `--without-threads` that I used to overcome some problem in the build,
the other 2 settings I copied from Debian package build settings.

I will look to rebuild removing `--without-threads`

I used 3 of the debian patches
https://sources.debian.org/patches/cdo/2.0.0~rc1-1/

32-bit.patch
deregister.patch
fortran-fix.patch

plus one to allow to the build of libcdi as shared library

The CDO packages are splitted in 4 as typical for this type of programs:
cdo cdo-devel libcdo0 libcdo-fortran0

$ cygcheck -l cdo
/usr/bin/cdi.exe
/usr/bin/cdo.exe
/usr/share/doc/cdo/AUTHORS
/usr/share/doc/cdo/ChangeLog
/usr/share/doc/cdo/COPYING
/usr/share/doc/cdo/NEWS
/usr/share/doc/cdo/README

$ cygcheck -l cdo-devel
/usr/lib/libcdi.dll.a
/usr/lib/libcdi_f2003.dll.a
/usr/lib/pkgconfig/cdi.pc
/usr/lib/pkgconfig/cdi_f2003.pc
/usr/include/cdi.h
/usr/include/cdi.inc
/usr/include/mo_cdi.mod

$ cygcheck -l libcdo0
/usr/bin/cygcdi-0.dll

$ cygcheck -l libcdo-fortran0
/usr/bin/cygcdi_f2003-0.dll

The full Cygwin Source code is at
http://matzeri.altervista.org/x86_64/release/cdo/cdo-2.0.0-0.1-src.tar.xz

It includes the build script fro cygport and all the patch used.

$ tar -tf cdo-2.0.0-0.1-src.tar.xz
cdo-2.0.0-0.1.src/
cdo-2.0.0-0.1.src/32-bit.patch
cdo-2.0.0-0.1.src/cdo-2.0.0-0.1.src.patch
cdo-2.0.0-0.1.src/cdo-2.0.0rc1.tar.gz
cdo-2.0.0-0.1.src/cdo.cygport
cdo-2.0.0-0.1.src/deregister.patch
cdo-2.0.0-0.1.src/fortran-fix.patch

Marco Atzeri wrote in RE: CDO package for Cygwin:

The full Cygwin Source code is at
http://matzeri.altervista.org/x86_64/release/cdo/cdo-2.0.0-0.1-src.tar.xz

updated dropping `--without-threads`
http://matzeri.altervista.org/x86_64/release/cdo/cdo-2.0.0-0.2-src.tar.xz

Maybe one more word regarding debian packages: Although it's in general a good idea to compare building with a long standing and stable distro like Debian, one has to consider the special requirements of Debian,too.

Debian has the habit of cracking software package into pieces that are not designed by the original software developers (-dev versions of libraries, -doc and -dev packages of scripting languages). you can see that all over the place when browsing the Debian package system. This is also the case for CDO, although CDO clearly comes with the CDI packages into one tar-bar. the reason behind this is the ability to limit the installation size to the very minimum. Debian supports very different hardware platforms and many of them occur in systems with very little resources. Therefor it would not make sense to have large packages that provide every functionality. Other distros do it differently: Arch linux use large packages, because they focus on x86-64 where disk space and compute performance or usually no issue.

I personally see cygwin more in the region of a PC-Specialized distro where there is no need to break packages in tiny pieces. The CDO package should IMO lead to only a single CDO binary installed. no CDI, no man-page. The CDI binary is more a development tool that something that needs to be installed for users. in terms of features it could very well be the case that CDO uses a feature which is not part of a fixed CDI release. for example: There is currently no public CDI-1.9.10, but this version is used in CDO-1.9.10. Sometimes CDO needs a fix you want to provide for the users, but in the context of the CDI library it deserves more attention and hence needs more time. So by splitting CDI and CDO in terms of packaging, you might limit the functionality of CDO for doing so. And I personally don't see the need for that regarding cygwin.

ok - just my thoughts ;-)

cheers + a good week
ralf

I installed the test packages from http://matzeri.altervista.org/.

 % cdo --version
Climate Data Operators version 2.0.0rc1 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-cygwin
CXX Compiler: g++ -std=gnu++14 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protectortrong --param=ssp-buffer-size=4 -fdebug-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/build=/usr/src/debug/cdo-2.0.0-2 -fdebug-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/src/cdo-2.0.0rc1=/usr/src/debug/cdo-2.0.0-0.2 -D_GNU_SOURCE -penmp -pthread
CXX version : g++ (GCC) 10.2.0
C Compiler: gcc -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --paramsp-buffer-size=4 -fdebug-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/build=/usr/src/debug/cdo-2.0.0-0.2 -fdebug-prex-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/src/cdo-2.0.0rc1=/usr/src/debug/cdo-2.0.0-0.2 -fopenmp -pthread -pthread
C version : gcc (GCC) 10.2.0
F77 Compiler: gfortran -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong param=ssp-buffer-size=4 -fdebug-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/build=/usr/src/debug/cdo-2.0.0-0.2 -fdeg-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/src/cdo-2.0.0rc1=/usr/src/debug/cdo-2.0.0-0.2
F77 version : GNU Fortran (GCC) 10.2.0
Features: 15GB 8threads C++14 OpenMP45 Fortran PTHREADS HDF5 NC4/HDF5 OPeNDAP SZ UDUNITS2 PROJ XML2 CURL FFTW3 SSE2
Libraries: HDF5/1.10.7 proj/6.3.1 xml2/2.9.10 curl/7.76.1
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
     CDI library version : 2.0.0rc1
 cgribex library version : 1.9.5
 ecCodes library version : 2.21.0
  NetCDF library version : 4.8.0 of Apr 18 2021 22:26:37 $
    hdf5 library version : 1.10.7
    exse library version : 1.4.2
    FILE library version : 1.9.1

Note that SZ is listed in Features. There is a problem with szip compression.

Using the NetCDF-4 file from the cdo-1.9 issue:

 % cdo -z zip4 copy topo.nc4 topo_zip4.nc4
cdo    copy: Processed 259200 values from 1 variable over 1 timestep [0.07s 21MB].
 % cdo -z szip copy topo.nc4 topo_szip.nc4
Warning (cdfDefVarSzip): Szip compression failed, NetCDF4/szlib not available!
cdo    copy: Processed 259200 values from 1 variable over 1 timestep [0.05s 19MB].
 % ls -l *.nc4
-rw-r--r-- 1 X 1056144 Mar 27 12:18 topo.nc4
-rw-r--r-- 1 X 1056018 May  6 09:51 topo_szip.nc4
-rw-r--r-- 1 X  553459 May  6 09:51 topo_zip4.nc4

My understanding is that netcdf should get szip thru hdf5, but export.and
or commerial use restrictions prevent open distributions like cygwin and
linux distros from supplying szip compression. Does that mean we have to
build the libraries and CDO locally to use szip compression?

George White wrote in RE: CDO package for Cygwin:

I installed the test packages from http://matzeri.altervista.org/.

[...]

Note that SZ is listed in Features. There is a problem with szip compression.

Using the NetCDF-4 file from the cdo-1.9 issue:

[...]

My understanding is that netcdf should get szip thru hdf5, but export.and
or commerial use restrictions prevent open distributions like cygwin and
linux distros from supplying szip compression. Does that mean we have to
build the libraries and CDO locally to use szip compression?

The current hdf5 package on Cygwin was built with libsz

$ cygcheck -c libhdf5_103
Cygwin Package Information
Package Version Status
libhdf5_103 1.10.7-2 OK

$ cygcheck -l libhdf5_103
/usr/bin/cyghdf5-103.dll

$ cygcheck /usr/bin/cyghdf5-103.dll
D:\cygwin64\bin\cyghdf5-103.dll
D:\cygwin64\bin\cygwin1.dll
C:\WINDOWS\system32\KERNEL32.dll
C:\WINDOWS\system32\ntdll.dll
C:\WINDOWS\system32\KERNELBASE.dll
D:\cygwin64\bin\cygsz-2.dll <- Here
D:\cygwin64\bin\cygaec-0.dll
D:\cygwin64\bin\cygz.dll

I will look if also netcdf need a rebuild with SZ

Marco Atzeri wrote in RE: CDO package for Cygwin:

libSZ is provided by https://gitlab.dkrz.de/k202009/libaec

Now that you mention it, I remember that libaec had been used to replace libSZ. I see:

% ldd /usr/bin/cdo.exe | grep -e 'cyg.*df.*dll'
        cyghdf5-103.dll => /usr/bin/cyghdf5-103.dll (0x3e81f0000)
        cygnetcdf-19.dll => /usr/bin/cygnetcdf-19.dll (0x3dc840000)
        cyghdf5_hl-100.dll => /usr/bin/cyghdf5_hl-100.dll (0x3e8160000)
        cygmfhdf-0.dll => /usr/bin/cygmfhdf-0.dll (0x3dd290000)
        cygdf-0.dll => /usr/bin/cygdf-0.dll (0x3ef710000)
% ldd /usr/bin/cdo.exe | grep aec
        cygaec-0.dll => /usr/bin/cygaec-0.dll (0x3f6d80000)
% ldd /usr/bin/cyghdf5-103.dll | grep aec
        cygaec-0.dll => /usr/bin/cygaec-0.dll (0x3f6d80000)
% ldd /usr/bin/cygnetcdf-19.dll | grep aec
        cygaec-0.dll => /usr/bin/cygaec-0.dll (0x3f6d80000)
% ldd /usr/bin/cyghdf5_hl-100.dll | grep aec
        cygaec-0.dll => /usr/bin/cygaec-0.dll (0x3f6d80000)

Why isn't it being used? Is there some missing configuration tweak or do I have a wrong .dll?

George White wrote in RE: CDO package for Cygwin:

Now that you mention it, I remember that libaec had been used to replace libSZ. I see:

Why isn't it being used? Is there some missing configuration tweak or do I have a wrong .dll?

It seems NetCDF check the libsz presence at build time, not at run time;
I built netcdf before adding libsz to HDF5, so it is not aware.

I am rebuilding NetCDF and I will let you know.

After rebuilding NetCDF and CDO, it is now working.

$ cdo -z szip copy topo.nc4 topo_szip.nc4
cdo copy: Processed 259200 values from 1 variable over 1 timestep [2.59s 16MB].

$ cdo showformat topo.nc4
NetCDF4
cdo showformat: Processed 1 variable [0.45s 13MB].

$ cdo showformat topo_szip.nc4
NetCDF4 szip
cdo showformat: Processed 1 variable [0.01s 13MB].

$ ls -1sh topo.nc4 topo_szip.nc4
1.1M topo.nc4
804K topo_szip.nc4

let me rebuild the 32bit version, and I will upload both.

netcdf-4.8.0-3 is up on cygwin.com
allow some time for mirror updates

cdo-2.0.0-0.3 is up on http://matzeri.altervista.org/

Thanks. Both zip and sz compression are working:

$ ls -l *.nc4
-rw-r--r-- 1 WhiteG Domain Users 1055895 May  8 12:09 topo.nc4
-rw-r--r-- 1 WhiteG Domain Users  819946 May  8 12:25 topo_cdo-z_szip.nc4
-rw-r--r-- 1 WhiteG Domain Users  553408 May  8 12:09 topo_cdo-z_zip4.nc4
-rw-r--r-- 1 WhiteG Domain Users  674239 May  8 12:09 topo_nccopy-d1.nc4
-rw-r--r-- 1 WhiteG Domain Users  548421 May  8 12:09 topo_nccopy-d1s.nc4

(I get much better compression ratios for the products I'm actually using).

Note: The current NASA OCSSW software uses hdf5-1.12.0 with netcdf-4.7.3.

From "hdf5-1.10.7-RELEASE.txt": "HDF5 binaries provided are fully
tested with ZLIB and the free Open Source SZIP successor Libaec (with
BSD license)."

George White wrote in RE: CDO package for Cygwin:

Thanks. Both zip and sz compression are working:

Thanks for the test.

(I get much better compression ratios for the products I'm actually using).

Do you mean my build is better, or much inferior to what you are using ?

Note: The current NASA OCSSW software uses hdf5-1.12.0 with netcdf-4.7.3.

I decided to not pack hdf5-1.12.x as netcdf-4.8.0 expects a HDF5 with API
1.8 or 1.10. It does not work yet with 1.12 API.
I saw little sense to build 1.12 with 1.10 API.

From "hdf5-1.10.7-RELEASE.txt": "HDF5 binaries provided are fully
tested with ZLIB and the free Open Source SZIP successor Libaec (with
BSD license)."

I hope that is true also for cygwin build ;-)

orig has 1 year of (12 monthly) netcdf4 files created
using R. zip4 and SZ are the cdo compressed files:

% du -sm orig zip4 SZ
56      orig
18      zip4
12      SZ

CDO 2.0.0rc1 has been added to the Cygwin distribution,
for both 32 and 64 bit architecture

https://sourceware.org/pipermail/cygwin-announce/2021-May/010059.html

(as HDF4 is not available on 32bit, I suggest the usage of the 64bit platform)

Cygwin package updated to 2.0.2

https://sourceware.org/pipermail/cygwin-announce/2021-December/010377.html

at the same time also updated hdf5, netcdf and eccodes

https://sourceware.org/pipermail/cygwin-announce/2021-December/date.html

this is great news - thx again for your support Marco!!

and happy new year btw

Many thanks -- the new version handles simple tests with NASA OBPG mapped NetCDF4-CF files. After updating to Fedora 35 I have to rebuild CDO to use libproj.so.22.2.0 (or configure it to use NASA OCSSW libraries), but meanwhile I can use Cygwin.

Being a new users, I'm not sure of where to send this. I just downloaded cdo-2.2.0. I'm running Windows 11, with cygwin. As per the instructions in the INSTALL file, I ran the ./configure script. I needed to get the C++ and Python packages for cygwin, which I did, and the configure script ran with no apparent problems. Next, I ran the make file. It ran for several minutes, until I got this:

make  all-am
make[2]: Entering directory '/home/climate/CDO/cdo-2.2.0/src'
g++ -DHAVE_CONFIG_H -I.  -I../libcdi/src -I../src/mpim_grid -DYAC_FOR_CDO -DCDI_SIZE_TYPE=size_t  -g -O2 -fopenmp -pthread -MT cdo-cdo.o -MD -MP -MF .deps/cdo-cdo.Tpo -c -o cdo-cdo.o `test -f 'cdo.cc' || echo './'`cdo.cc
cdo.cc: In function ‘int cdo_feenableexcept(int)’:
cdo.cc:111:10: error: ‘feenableexcept’ was not declared in this scope; did you mean ‘feraiseexcept’?
  111 |   return feenableexcept(excepts);
      |          ^~~~~~~~~~~~~~
      |          feraiseexcept
make[2]: *** [Makefile:2650: cdo-cdo.o] Error 1
make[2]: Leaving directory '/home/climate/CDO/cdo-2.2.0/src'
make[1]: *** [Makefile:1025: all] Error 2
make[1]: Leaving directory '/home/climate/CDO/cdo-2.2.0/src'
make: *** [Makefile:508: all-recursive] Error 1

Perhaps someone more familiar with this package can tell me what I need to do to get this project up and running.

Thank you in advance,

Michael

hi Michael!

When using cygwin there is not need to install CDO manually. cygwin comes with pre-build CDO binary. just use the latest setup.exe and cdo should be ready to install with it.

cheers
ralf

Thank you! I got a fix for the problem, and it's happily compiling away. I'll know for next time, though.

https://sourceware.org/pipermail/cygwin-announce/2024-February/011540.html

Updated: cdo-2.3.0-1

just uploaded on Cygwin server.

Regards Marco

thx a lot, Marco!!

This might be useful. You can run CDO v2.4 as a portable app on Windows if you copy the following DLLs from your current CYGWIN installation:
cygaec-0.dll
cygbrotlicommon-1.dll
cygbrotlidec-1.dll
cygbz2-1.dll
cygcdi-0.dll
cygcom_err-2.dll
cygcrypto-1.0.0.dll
cygcrypto-1.1.dll
cygcrypto-3.dll
cygcurl-4.dll
cygdeflate-0.dll
cygdf-0.dll
cygeccodes.dll
cygexpat-1.dll
cygfftw3-3.dll
cyggcc_s-seh-1.dll
cyggcrypt-20.dll
cyggomp-1.dll
cyggpg-error-0.dll
cyggsasl-18.dll
cyggssapi_krb5-2.dll
cyghdf5-200.dll
cyghdf5_hl-200.dll
cygiconv-2.dll
cygidn-12.dll
cygidn2-0.dll
cygintl-8.dll
cygjasper-4.dll
cygjbig-2.dll
cygjpeg-8.dll
cygk5crypto-3.dll
cygkrb5-3.dll
cygkrb5support-0.dll
cyglber-2.dll
cygldap-2.dll
cyglzma-5.dll
cygmfhdf-0.dll
cygnetcdf-19.dll
cygnghttp2-14.dll
cygntlm-0.dll
cygopenjpeg-1.dll
cygpng16-16.dll
cygproj-25.dll
cygpsl-5.dll
cygsasl2-3.dll
cygsharpyuv-0.dll
cygsqlite3-0.dll
cygssh2-1.dll
cygssl-3.dll
cygstdc++-6.dll
cygsz-2.dll
cygtiff-7.dll
cygudunits2-0.dll
cygunistring-5.dll
cyguuid-1.dll
cygwebp-7.dll
cygwin1.dll
cygxml2-2.dll
cygz.dll
cygzip-5.dll
cygzstd-1.dll

It's quite convenient to have in on an external SSD to use with different PCs/laptops.

thx! This is indeed very good to know

cheers
ralf