Project

General

Profile

CDO package for Cygwin

Added by Marco Atzeri almost 3 years ago

Good Day,
I am the current Cygwin package maintainer for HDF4, HDF5, NETCDF and few more

One user asked my help to build CDO with ecCodes, so after some work,
I have built the package for both the 32bit and 64bit version of Cygwin of

eccodes-2.21.0-1
nco-4.9.8-1
cdo-2.0.0-0.1 (based on your last Release Candidate)

I am planning to upload the packages on the Cygwin distribution
in the coming weeks, and I will appreciate some test and/or feedback.

The packages are available for test at http://matzeri.altervista.org/

The CDO program and library depends on the following libraries, so please
install them before testing the program

libcurl4-7.76.1-1
libfftw3_3-3.3.9-1
libgcc1-10.2.0-1
libgomp1-10.2.0-1
libhdf5_103-1.10.7-2
libnetcdf19-4.8.0-2
libproj15-6.3.1-1
libstdc++6-10.2.0-1
libsz2-1.0.4-1
libudunits0-2.2.27.6-1
libuuid1-2.33.1-2
libxml2-2.9.10-2

libaec0-1.0.4-1
libgfortran5-10.2.0-1
libjasper4-2.0.14-1
libopenjpeg1-1.5.2-3
libpng16-1.6.37-1

These are available on my repository
libeccodes-2.21.0-1
libeccodes-data-2.21.0-1

The package is currently configured with

--enable-shared --disable-static --with-szlib --with-zlib --with-hdf5 --with-netcdf --with-udunits2 --with-eccodes --with-fftw3 --with-proj --with-curl --with-libxml2 --without-threads --enable-cdi-lib --enable-iso-c-interface --enable-hirlam-extensions

Testsuite summary for cdo 2.0.0rc1 ============================================================================
  1. TOTAL: 614
  2. PASS: 537
  3. SKIP: 71
  4. XFAIL: 0
  5. FAIL: 6
  6. XPASS: 0
  7. ERROR: 0
$ grep FAIL cdo-2.0.0-0.1-check.log
FAIL: test_f2003
FAIL: MapReduce.test 1 - reducegrid on grid r18x9
FAIL: MapReduce.test 2 - reducegrid on grid icon_cell
FAIL: Remapgrid.test 4 - remapbic global_2 to global_5
FAIL: Remapgrid.test 6 - remapcon2 global_2 to global_5
FAIL: Remapgrid.test 12 - remapbic global_5 to global_2
FAIL: Remapgrid.test 14 - remapcon2 global_5 to global_2
  1. XFAIL: 0
  2. FAIL: 6

Regards
Marco Atzeri


Replies (23)

RE: CDO package for Cygwin - Added by Ralf Mueller almost 3 years ago

hi Marco!

Brandon (author of https://code.mpimet.mpg.de/issues/10069) wanted to work on this, too. so maybe you get in contact with each other.

The dependencies look normal to me, but your configure call has some flaws IMO:

  • --without-threads this is obsolete, in fact it will turn CDO into a single-thread application, which is not what people want. please, drop this options
  • --enable-cdi-lib CDO comes with its on CDI code out-of-the-box. Sometimes this code is not a CDI-release on its own, so you will not be able to link every CDO release to a certain CDI release. This configure switch only exist to have the option to build CDO against an existing CDI, but it should not be taken as the default way to build CDO
  • --enable-iso-c-interface this is a special CDI option. CDO does not need that switch. The background is: CDI is a libeary written in C but also offers a Fortran interface. With this switch you can tweak the way the fortran interface is build. CDO is plain C++, so CDI does not need to provide the Fortran at all.

Please adjust the options and run the tests again.

thx in advance
ralf

RE: CDO package for Cygwin - Added by Marco Atzeri almost 3 years ago

Hi Ralf,

except `--without-threads` that I used to overcome some problem in the build,
the other 2 settings I copied from Debian package build settings.

I will look to rebuild removing `--without-threads`

I used 3 of the debian patches
https://sources.debian.org/patches/cdo/2.0.0~rc1-1/

32-bit.patch
deregister.patch
fortran-fix.patch

plus one to allow to the build of libcdi as shared library

The CDO packages are splitted in 4 as typical for this type of programs:
cdo cdo-devel libcdo0 libcdo-fortran0

$ cygcheck -l cdo
/usr/bin/cdi.exe
/usr/bin/cdo.exe
/usr/share/doc/cdo/AUTHORS
/usr/share/doc/cdo/ChangeLog
/usr/share/doc/cdo/COPYING
/usr/share/doc/cdo/NEWS
/usr/share/doc/cdo/README

$ cygcheck -l cdo-devel
/usr/lib/libcdi.dll.a
/usr/lib/libcdi_f2003.dll.a
/usr/lib/pkgconfig/cdi.pc
/usr/lib/pkgconfig/cdi_f2003.pc
/usr/include/cdi.h
/usr/include/cdi.inc
/usr/include/mo_cdi.mod

$ cygcheck -l libcdo0
/usr/bin/cygcdi-0.dll

$ cygcheck -l libcdo-fortran0
/usr/bin/cygcdi_f2003-0.dll

The full Cygwin Source code is at
http://matzeri.altervista.org/x86_64/release/cdo/cdo-2.0.0-0.1-src.tar.xz

It includes the build script fro cygport and all the patch used.

$ tar -tf cdo-2.0.0-0.1-src.tar.xz
cdo-2.0.0-0.1.src/
cdo-2.0.0-0.1.src/32-bit.patch
cdo-2.0.0-0.1.src/cdo-2.0.0-0.1.src.patch
cdo-2.0.0-0.1.src/cdo-2.0.0rc1.tar.gz
cdo-2.0.0-0.1.src/cdo.cygport
cdo-2.0.0-0.1.src/deregister.patch
cdo-2.0.0-0.1.src/fortran-fix.patch

RE: CDO package for Cygwin - Added by Ralf Mueller almost 3 years ago

Maybe one more word regarding debian packages: Although it's in general a good idea to compare building with a long standing and stable distro like Debian, one has to consider the special requirements of Debian,too.

Debian has the habit of cracking software package into pieces that are not designed by the original software developers (-dev versions of libraries, -doc and -dev packages of scripting languages). you can see that all over the place when browsing the Debian package system. This is also the case for CDO, although CDO clearly comes with the CDI packages into one tar-bar. the reason behind this is the ability to limit the installation size to the very minimum. Debian supports very different hardware platforms and many of them occur in systems with very little resources. Therefor it would not make sense to have large packages that provide every functionality. Other distros do it differently: Arch linux use large packages, because they focus on x86-64 where disk space and compute performance or usually no issue.

I personally see cygwin more in the region of a PC-Specialized distro where there is no need to break packages in tiny pieces. The CDO package should IMO lead to only a single CDO binary installed. no CDI, no man-page. The CDI binary is more a development tool that something that needs to be installed for users. in terms of features it could very well be the case that CDO uses a feature which is not part of a fixed CDI release. for example: There is currently no public CDI-1.9.10, but this version is used in CDO-1.9.10. Sometimes CDO needs a fix you want to provide for the users, but in the context of the CDI library it deserves more attention and hence needs more time. So by splitting CDI and CDO in terms of packaging, you might limit the functionality of CDO for doing so. And I personally don't see the need for that regarding cygwin.

ok - just my thoughts ;-)

cheers + a good week
ralf

RE: CDO package for Cygwin - Added by George White almost 3 years ago

I installed the test packages from http://matzeri.altervista.org/.

 % cdo --version
Climate Data Operators version 2.0.0rc1 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-cygwin
CXX Compiler: g++ -std=gnu++14 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protectortrong --param=ssp-buffer-size=4 -fdebug-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/build=/usr/src/debug/cdo-2.0.0-2 -fdebug-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/src/cdo-2.0.0rc1=/usr/src/debug/cdo-2.0.0-0.2 -D_GNU_SOURCE -penmp -pthread
CXX version : g++ (GCC) 10.2.0
C Compiler: gcc -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --paramsp-buffer-size=4 -fdebug-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/build=/usr/src/debug/cdo-2.0.0-0.2 -fdebug-prex-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/src/cdo-2.0.0rc1=/usr/src/debug/cdo-2.0.0-0.2 -fopenmp -pthread -pthread
C version : gcc (GCC) 10.2.0
F77 Compiler: gfortran -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong param=ssp-buffer-size=4 -fdebug-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/build=/usr/src/debug/cdo-2.0.0-0.2 -fdeg-prefix-map=/pub/devel/cdo/cdo-2.0.0-0.2.x86_64/src/cdo-2.0.0rc1=/usr/src/debug/cdo-2.0.0-0.2
F77 version : GNU Fortran (GCC) 10.2.0
Features: 15GB 8threads C++14 OpenMP45 Fortran PTHREADS HDF5 NC4/HDF5 OPeNDAP SZ UDUNITS2 PROJ XML2 CURL FFTW3 SSE2
Libraries: HDF5/1.10.7 proj/6.3.1 xml2/2.9.10 curl/7.76.1
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
     CDI library version : 2.0.0rc1
 cgribex library version : 1.9.5
 ecCodes library version : 2.21.0
  NetCDF library version : 4.8.0 of Apr 18 2021 22:26:37 $
    hdf5 library version : 1.10.7
    exse library version : 1.4.2
    FILE library version : 1.9.1

Note that SZ is listed in Features. There is a problem with szip compression.

Using the NetCDF-4 file from the cdo-1.9 issue:
 % cdo -z zip4 copy topo.nc4 topo_zip4.nc4
cdo    copy: Processed 259200 values from 1 variable over 1 timestep [0.07s 21MB].
 % cdo -z szip copy topo.nc4 topo_szip.nc4
Warning (cdfDefVarSzip): Szip compression failed, NetCDF4/szlib not available!
cdo    copy: Processed 259200 values from 1 variable over 1 timestep [0.05s 19MB].
 % ls -l *.nc4
-rw-r--r-- 1 X 1056144 Mar 27 12:18 topo.nc4
-rw-r--r-- 1 X 1056018 May  6 09:51 topo_szip.nc4
-rw-r--r-- 1 X  553459 May  6 09:51 topo_zip4.nc4

My understanding is that netcdf should get szip thru hdf5, but export.and
or commerial use restrictions prevent open distributions like cygwin and
linux distros from supplying szip compression. Does that mean we have to
build the libraries and CDO locally to use szip compression?

RE: CDO package for Cygwin - Added by Marco Atzeri almost 3 years ago

George White wrote in RE: CDO package for Cygwin:

I installed the test packages from http://matzeri.altervista.org/.

[...]

Note that SZ is listed in Features. There is a problem with szip compression.

Using the NetCDF-4 file from the cdo-1.9 issue:

[...]

My understanding is that netcdf should get szip thru hdf5, but export.and
or commerial use restrictions prevent open distributions like cygwin and
linux distros from supplying szip compression. Does that mean we have to
build the libraries and CDO locally to use szip compression?

The current hdf5 package on Cygwin was built with libsz

$ cygcheck -c libhdf5_103
Cygwin Package Information
Package Version Status
libhdf5_103 1.10.7-2 OK

$ cygcheck -l libhdf5_103
/usr/bin/cyghdf5-103.dll

$ cygcheck /usr/bin/cyghdf5-103.dll
D:\cygwin64\bin\cyghdf5-103.dll
D:\cygwin64\bin\cygwin1.dll
C:\WINDOWS\system32\KERNEL32.dll
C:\WINDOWS\system32\ntdll.dll
C:\WINDOWS\system32\KERNELBASE.dll
D:\cygwin64\bin\cygsz-2.dll <- Here
D:\cygwin64\bin\cygaec-0.dll
D:\cygwin64\bin\cygz.dll

I will look if also netcdf need a rebuild with SZ

RE: CDO package for Cygwin - Added by George White almost 3 years ago

Now that you mention it, I remember that libaec had been used to replace libSZ. I see:

% ldd /usr/bin/cdo.exe | grep -e 'cyg.*df.*dll'
        cyghdf5-103.dll => /usr/bin/cyghdf5-103.dll (0x3e81f0000)
        cygnetcdf-19.dll => /usr/bin/cygnetcdf-19.dll (0x3dc840000)
        cyghdf5_hl-100.dll => /usr/bin/cyghdf5_hl-100.dll (0x3e8160000)
        cygmfhdf-0.dll => /usr/bin/cygmfhdf-0.dll (0x3dd290000)
        cygdf-0.dll => /usr/bin/cygdf-0.dll (0x3ef710000)
% ldd /usr/bin/cdo.exe | grep aec
        cygaec-0.dll => /usr/bin/cygaec-0.dll (0x3f6d80000)
% ldd /usr/bin/cyghdf5-103.dll | grep aec
        cygaec-0.dll => /usr/bin/cygaec-0.dll (0x3f6d80000)
% ldd /usr/bin/cygnetcdf-19.dll | grep aec
        cygaec-0.dll => /usr/bin/cygaec-0.dll (0x3f6d80000)
% ldd /usr/bin/cyghdf5_hl-100.dll | grep aec
        cygaec-0.dll => /usr/bin/cygaec-0.dll (0x3f6d80000)

Why isn't it being used? Is there some missing configuration tweak or do I have a wrong .dll?

RE: CDO package for Cygwin - Added by Marco Atzeri almost 3 years ago

George White wrote in RE: CDO package for Cygwin:

Now that you mention it, I remember that libaec had been used to replace libSZ. I see:

Why isn't it being used? Is there some missing configuration tweak or do I have a wrong .dll?

It seems NetCDF check the libsz presence at build time, not at run time;
I built netcdf before adding libsz to HDF5, so it is not aware.

I am rebuilding NetCDF and I will let you know.

RE: CDO package for Cygwin - Added by Marco Atzeri almost 3 years ago

After rebuilding NetCDF and CDO, it is now working.

$ cdo -z szip copy topo.nc4 topo_szip.nc4
cdo copy: Processed 259200 values from 1 variable over 1 timestep [2.59s 16MB].

$ cdo showformat topo.nc4
NetCDF4
cdo showformat: Processed 1 variable [0.45s 13MB].

$ cdo showformat topo_szip.nc4
NetCDF4 szip
cdo showformat: Processed 1 variable [0.01s 13MB].

$ ls -1sh topo.nc4 topo_szip.nc4
1.1M topo.nc4
804K topo_szip.nc4

let me rebuild the 32bit version, and I will upload both.

RE: CDO package for Cygwin - Added by Marco Atzeri almost 3 years ago

netcdf-4.8.0-3 is up on cygwin.com
allow some time for mirror updates

cdo-2.0.0-0.3 is up on http://matzeri.altervista.org/

RE: CDO package for Cygwin - Added by George White almost 3 years ago

Thanks. Both zip and sz compression are working:

$ ls -l *.nc4
-rw-r--r-- 1 WhiteG Domain Users 1055895 May  8 12:09 topo.nc4
-rw-r--r-- 1 WhiteG Domain Users  819946 May  8 12:25 topo_cdo-z_szip.nc4
-rw-r--r-- 1 WhiteG Domain Users  553408 May  8 12:09 topo_cdo-z_zip4.nc4
-rw-r--r-- 1 WhiteG Domain Users  674239 May  8 12:09 topo_nccopy-d1.nc4
-rw-r--r-- 1 WhiteG Domain Users  548421 May  8 12:09 topo_nccopy-d1s.nc4

(I get much better compression ratios for the products I'm actually using).

Note: The current NASA OCSSW software uses hdf5-1.12.0 with netcdf-4.7.3.

From "hdf5-1.10.7-RELEASE.txt": "HDF5 binaries provided are fully
tested with ZLIB and the free Open Source SZIP successor Libaec (with
BSD license)."

RE: CDO package for Cygwin - Added by Marco Atzeri almost 3 years ago

George White wrote in RE: CDO package for Cygwin:

Thanks. Both zip and sz compression are working:

Thanks for the test.

(I get much better compression ratios for the products I'm actually using).

Do you mean my build is better, or much inferior to what you are using ?

Note: The current NASA OCSSW software uses hdf5-1.12.0 with netcdf-4.7.3.

I decided to not pack hdf5-1.12.x as netcdf-4.8.0 expects a HDF5 with API
1.8 or 1.10. It does not work yet with 1.12 API.
I saw little sense to build 1.12 with 1.10 API.

From "hdf5-1.10.7-RELEASE.txt": "HDF5 binaries provided are fully
tested with ZLIB and the free Open Source SZIP successor Libaec (with
BSD license)."

I hope that is true also for cygwin build ;-)

RE: CDO package for Cygwin - Added by George White almost 3 years ago

orig has 1 year of (12 monthly) netcdf4 files created
using R. zip4 and SZ are the cdo compressed files:

% du -sm orig zip4 SZ
56 orig
18 zip4
12 SZ

RE: CDO package for Cygwin - Added by Marco Atzeri almost 3 years ago

CDO 2.0.0rc1 has been added to the Cygwin distribution,
for both 32 and 64 bit architecture

https://sourceware.org/pipermail/cygwin-announce/2021-May/010059.html

(as HDF4 is not available on 32bit, I suggest the usage of the 64bit platform)

RE: CDO package for Cygwin - Added by Ralf Mueller about 2 years ago

this is great news - thx again for your support Marco!!

and happy new year btw

RE: CDO package for Cygwin - Added by George White about 2 years ago

Many thanks -- the new version handles simple tests with NASA OBPG mapped NetCDF4-CF files. After updating to Fedora 35 I have to rebuild CDO to use libproj.so.22.2.0 (or configure it to use NASA OCSSW libraries), but meanwhile I can use Cygwin.

RE: CDO package for Cygwin - Added by Michael Purves 10 months ago

Being a new users, I'm not sure of where to send this. I just downloaded cdo-2.2.0. I'm running Windows 11, with cygwin. As per the instructions in the INSTALL file, I ran the ./configure script. I needed to get the C++ and Python packages for cygwin, which I did, and the configure script ran with no apparent problems. Next, I ran the make file. It ran for several minutes, until I got this:

make  all-am
make[2]: Entering directory '/home/climate/CDO/cdo-2.2.0/src'
g++ -DHAVE_CONFIG_H -I.  -I../libcdi/src -I../src/mpim_grid -DYAC_FOR_CDO -DCDI_SIZE_TYPE=size_t  -g -O2 -fopenmp -pthread -MT cdo-cdo.o -MD -MP -MF .deps/cdo-cdo.Tpo -c -o cdo-cdo.o `test -f 'cdo.cc' || echo './'`cdo.cc
cdo.cc: In function ‘int cdo_feenableexcept(int)’:
cdo.cc:111:10: error: ‘feenableexcept’ was not declared in this scope; did you mean ‘feraiseexcept’?
  111 |   return feenableexcept(excepts);
      |          ^~~~~~~~~~~~~~
      |          feraiseexcept
make[2]: *** [Makefile:2650: cdo-cdo.o] Error 1
make[2]: Leaving directory '/home/climate/CDO/cdo-2.2.0/src'
make[1]: *** [Makefile:1025: all] Error 2
make[1]: Leaving directory '/home/climate/CDO/cdo-2.2.0/src'
make: *** [Makefile:508: all-recursive] Error 1

Perhaps someone more familiar with this package can tell me what I need to do to get this project up and running.

Thank you in advance,

Michael

RE: CDO package for Cygwin - Added by Ralf Mueller 9 months ago

hi Michael!

When using cygwin there is not need to install CDO manually. cygwin comes with pre-build CDO binary. just use the latest setup.exe and cdo should be ready to install with it.

cheers
ralf

RE: CDO package for Cygwin - Added by Michael Purves 9 months ago

Thank you! I got a fix for the problem, and it's happily compiling away. I'll know for next time, though.

RE: CDO package for Cygwin - Added by Marco Atzeri about 2 months ago

https://sourceware.org/pipermail/cygwin-announce/2024-February/011540.html

Updated: cdo-2.3.0-1

just uploaded on Cygwin server.

Regards Marco

RE: CDO package for Cygwin - Added by Ralf Mueller about 2 months ago

thx a lot, Marco!!

    (1-23/23)