CDO

It would be easier to diagnose if you uploaded a sample file.

Hi Brendan,

Thanks for your help.
I have tested several nc files and cdo commands. I do not think the problem depends on the file.
However, here attached one of the netcdf files I have tested. I can open it in R and Panoply and I can explore it through a ncdump command, but when I try to do something in cdo, I get always the same 'Unknown Error'.
Thanks

I remember this problem when working with cygwin not too long ago. Unfortunately I don't remember what I did to prevent this. Possibly an update of cygwin, but you seem to have done that already. can you run ldd on the cdo.exe file?

Fabio Farinosi wrote:

I have tested several nc files and cdo commands. I do not think the problem depends on the file.

But it helps me see if my system has the same problem:

$ cdo -info sfcWind_EUR-11i_ECMWF-ERAINT_evaluation_r1i1p1_DHMZ-RegCM4-2_v1_mon_198901-199012.nccdo    info: Open failed on >sfcWind_EUR-11i_ECMWF-ERAINT_evaluation_r1i1p1_DHMZ-RegCM4-2_v1_mon_198901-199012.nc
             Unknown Error

I have a little more information and a work-around. CDO fails because your file is a netcdf4 classic:

]$ ncdump -k sfcWind_EUR-11i_ECMWF-ERAINT_evaluation_r1i1p1_DHMZ-RegCM4-2_v1_mon_198901-199012.nc
netCDF-4 classic model

C:\Users\detraceyb\Downloads>ncks -O -5 sfcWind_EUR-11i_ECMWF-ERAINT_evaluation_r1i1p1_DHMZ-RegCM4-2_v1_mon_198901-199012.nc test.nc

        1 file(s) moved.

$ cdo sinfo test.nc
   File format : NetCDF5
    -1 : Institut Source   T Steptype Levels Num    Points Num Dtype : Parameter ID
     1 : unknown  unknown  v instant       1   1    359448   1  F32  : -1
   Grid coordinates :
     1 : lonlat                   : points=359448 (881x408)
                              lon : -44.8125 to 65.1875 by 0.125 degrees_east
                              lat : 21.8125 to 72.6875 by 0.125 degrees_north
   Vertical coordinates :
     1 : height                   : levels=1  scalar
                           height : 10 m
   Time coordinate :  24 steps
     RefTime =  1949-12-01 00:00:00  Units = days  Calendar = proleptic_gregorian  Bounds = true
  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss
  1989-01-16 12:00:00  1989-02-15 00:00:00  1989-03-16 12:00:00  1989-04-16 00:00:00
  1989-05-16 12:00:00  1989-06-16 00:00:00  1989-07-16 12:00:00  1989-08-16 12:00:00
  1989-09-16 00:00:00  1989-10-16 12:00:00  1989-11-16 00:00:00  1989-12-16 12:00:00
  1990-01-16 12:00:00  1990-02-15 00:00:00  1990-03-16 12:00:00  1990-04-16 00:00:00
  1990-05-16 12:00:00  1990-06-16 00:00:00  1990-07-16 12:00:00  1990-08-16 12:00:00
  1990-09-16 00:00:00  1990-10-16 12:00:00  1990-11-16 00:00:00  1990-12-16 12:00:00
cdo    sinfo: Processed 1 variable over 24 timesteps [0.02s 16MB].

CDO fails on your file when I convert it using ncks conversion options: -4 (netcdf4) and -7 (netcdf4_classic)
You can download Cygwin binaries for NCO (http://nco.sourceforge.net/#Executables) and convert your files to netCDF3 64-bit data before processing with CDO. I suspect the real answer would be a newer version of CDO compiled with more recent libraries of hdf and netcdf. Better yet, install a Linux VM or WSL1. Then you can use the Linux version of anaconda to get access to CDO and/or NCO.

(Or WSL2...)

Thx Brendon! I wasn't sure how to go on with this. My impression is that Windows Linux subsystem is more stable when it comes to CDO, but since I do not regularly work with it, I didn't feel confident enough to comment on this.

Did you come to a similar conclusion?

Thanks a lot for your replies.
Unfortunately, on the office machines we have some software limitations and sometimes we are forced to use "suboptimal" tools (I see CDO is now updated to version 1.9.10, while for Cygwin stopped at 1.9.8).
As a matter of fact, working on a Unix environment (internal cluster) with the same files, CDO works flawlessly. I was just wondering why it did not work on my desktop. I'll try installing a WSL2 (if my system Admin lets me do that!) and use the most updated version.

Hahaha. Fabio, you must be a civil servant like I am, because I too know the meaning of software limitations and suboptimal tools!

As for WSL, I had success with WSL1 and cdo installed with anaconda. I suggested WSL2 because it is supposed to be faster, closer to the metal than WSL1, but I have not tested it. I have network access to Linux, but a windows craptop as terminal. I use cygwin cdo sometimes for convenience. I do not use WSL for CDO anymore, but did use it a little at the beginning of the plague.

I just run into the same problem, but on a Win10 machine. Updated cygwin last week and now cdo doesn't work properly any longer. It works on some files but not on others. And indeed, the problem seems to be with netcdf4/hdf. I attach a netcdf-2 file that I have created with cdo, and cdo works fine with this file (e.g. cdo sinfo x.nc). Hwoever, when I then try to convert the file to netcdf4 I get

$ cdo -f nc4c copy test_norcp.nc y.nc

Error (cdf_enddef): NetCDF: HDF error
HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
  #000: /cygdrive/e/cyg_pub/devel/hdf5/hdf5-1.10.2-1.x86_64/src/hdf5-1.10.2/src/H5A.c line 1669 in H5Aexists(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: /cygdrive/e/cyg_pub/devel/hdf5/hdf5-1.10.2-1.x86_64/src/hdf5-1.10.2/src/H5Gloc.c line 193 in H5G_loc(): invalid group ID
    major: Invalid arguments to routine
    minor: Bad value

Error (cdf_close): NetCDF: HDF error

It seems cdo expects the hdf library at /cygdrive/e/cyg_pub/devel/hdf5, but I don't have the hdf library there. Is it possible that the cdo release has been built with dynamic libraries and therefore doesn't work on other machines that are not identical? Could this be fixed somehow?

hi Klaus!

yes, CDO is build with shared libraries. you can check this with the ldd command. the problem is, that cygwin packages provide shared libs, but not static ones AFAIK.

Thanks a lot Ralf for the quick reply. Well, I don't know enough about what's under the hood of cygwin and cdo to dig deeper in this matter. However, I have found a workaround by installing cdo version 1.9.5. It's not nice to go back to an older version but it saves me a lot of headaches. And what's nice: I can use cdo-1.9.5 to convert nc4c to nc2 files, and after that cdo-1.9.7.1 works without problems. This saves me the trouble to install nco as it has been suggested otherwise.

However, in the long run I'd like to keep my cdo under cygwin up to date. What I don't understand is why the build for 1.9.5 results in a working copy of cdo, but not 1.9.7.1. What were the differences in the compilation of 1.95 and 1.9.7.1?

could you run ldd on the cdo-1.9.5 binary, please. The output could tell me, how this was build. and the output of

cdo -V

thx in advance
ralf

Here we go:

$ ./cdo.exe -V
Climate Data Operators version 1.9.5 (http://mpimet.mpg.de/cdo)
System: x86_64-unknown-cygwin
CXX Compiler: g++ -g -Wall -O2 -DPIC -pie -mwindows -fopenmp 
CXX version : g++ (GCC) 6.4.0
C Compiler: gcc -g -Wall -O2 -DPIC -pie -mwindows -fopenmp  
C version : gcc (GCC) 6.4.0
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (GCC) 6.4.0
Features: 7GB 4threads C++14 Fortran DATA PTHREADS OpenMP45 HDF5 NC4/HDF5/threadsafe OPeNDAP UDUNITS2 PROJ.4 SSE2
Libraries: HDF5/1.8.17 proj/5
Filetypes: srv ext ieg grb1 grb2 nc1 nc4 nc4c nc5 
     CDI library version : 1.9.5
 CGRIBEX library version : 1.9.1
GRIB_API library version : 1.16.0
  NetCDF library version : 4.4.0 of Jun 21 2016 15:28:18 $
    HDF5 library version : 1.8.17 threadsafe
    EXSE library version : 1.4.0
    FILE library version : 1.8.3

and

$ ldd ./cdo.exe
        ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x7ff8f3740000)
        KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x7ff8f1a0000
0)
        KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x7ff8f07
50000)
        cygcurl-4.dll => /usr/bin/cygcurl-4.dll (0x3d97c0000)
        cygstdc++-6.dll => /usr/bin/cygstdc++-6.dll (0x3aaf70000)
        cyggomp-1.dll => /usr/bin/cyggomp-1.dll (0x3ce770000)
        cyggcc_s-seh-1.dll => /usr/bin/cyggcc_s-seh-1.dll (0x3d4d80000)
        cygproj-13.dll => /usr/bin/cygproj-13.dll (0x3aeec0000)
        cyguuid-1.dll => /usr/bin/cyguuid-1.dll (0x3a92b0000)
        cygudunits2-0.dll => /usr/bin/cygudunits2-0.dll (0x3a9830000)
        cygz.dll => /usr/bin/cygz.dll (0x3a2730000)
        cygwin1.dll => /usr/bin/cygwin1.dll (0x180040000)
        cygwin1.dll => /usr/bin/cygwin1.dll (0xe30000)
        cygbrotlidec-1.dll => /usr/bin/cygbrotlidec-1.dll (0x3dfe00000)
        cygcrypto-1.1.dll => /usr/bin/cygcrypto-1.1.dll (0x3d9910000)
        cyggssapi_krb5-2.dll => /usr/bin/cyggssapi_krb5-2.dll (0x3cd2a0000)     
        cygidn2-0.dll => /usr/bin/cygidn2-0.dll (0x3bc450000)
        cyglber-2-4-2.dll => /usr/bin/cyglber-2-4-2.dll (0x3b87a0000)
        cygldap-2-4-2.dll => /usr/bin/cygldap-2-4-2.dll (0x3b86b0000)
        cygnghttp2-14.dll => /usr/bin/cygnghttp2-14.dll (0x3b4c70000)
        cygpsl-5.dll => /usr/bin/cygpsl-5.dll (0x3aeb90000)
        cygssh2-1.dll => /usr/bin/cygssh2-1.dll (0x3ab2e0000)
        cygssl-1.1.dll => /usr/bin/cygssl-1.1.dll (0x3ab1e0000)
        cygzstd-1.dll => /usr/bin/cygzstd-1.dll (0x3a1f50000)
        cygintl-8.dll => /usr/bin/cygintl-8.dll (0x3bc110000)
        cygexpat-1.dll => /usr/bin/cygexpat-1.dll (0x3d8250000)
        cygk5crypto-3.dll => /usr/bin/cygk5crypto-3.dll (0x3bb4c0000)
        cygbrotlicommon-1.dll => /usr/bin/cygbrotlicommon-1.dll (0x3dfe20000)   
        cygkrb5-3.dll => /usr/bin/cygkrb5-3.dll (0x3b8d20000)
        cygkrb5support-0.dll => /usr/bin/cygkrb5support-0.dll (0x3b8d00000)     
        cygcom_err-2.dll => /usr/bin/cygcom_err-2.dll (0x3d9f10000)
        cygiconv-2.dll => /usr/bin/cygiconv-2.dll (0x3c9910000)
        cygunistring-2.dll => /usr/bin/cygunistring-2.dll (0x3a94d0000)
        cygsasl2-3.dll => /usr/bin/cygsasl2-3.dll (0x3ac5b0000)
        cygcrypto-1.0.0.dll => /usr/bin/cygcrypto-1.0.0.dll (0x3d9b80000)       

thx a lot - obviously the netcdf,hdf5 and grib_api are linked statically. so the binary is independent of the cygwin packages.

meh - it's such a pain to install all dependencies ...

Ralf Mueller wrote:

thx a lot - obviously the netcdf,hdf5 and grib_api are linked statically. so the binary is independent of the cygwin packages.

meh - it's such a pain to install all dependencies ...

You're telling me... I still can't get cdo to compile under cygwin. Still trying. And I don't understand what's under the hood with cygwin with respect to static or shared libraries. What I think I know, is that if you build everything yourself from source and link everything statically, it works. This is a massive pain.

What I know from personal experience is that anyone using cygwin is doing so because they have no choice. Their ill-informed corporate masters will not let them install software under windows, and will not let them run Linux, Virtualbox, or WSL. They are stuck with a non-admin install of cygwin. I sympathize.