compression netcdf file with cdo version 2.4.0
Added by Andries De Vries about 1 month ago
Hello,
Previously, I used successfully the following command to compress netcdf files substantially:
cdo -f nc4c -z zip copy {ifile} {ofile}
Currently, cdo version 2.4.0 is installed by admin on our HPC. Compressing files with the above command results in the same file size, i.e. it doesn't work anymore. I do not know which previous cdo version was installed when this command worked, but on another cluster cdo version 1.9.6. is installed and this compression works well.
Do you have any recommendations? Is there an issue with cdo v.2.4.0? Is perhaps cdo not well installed as a module on the cluster for the purpose allowing compression? Or do I need to use another command with cdo 2.4.0 for compressing files?
Any help is much appreciated.
Best,
Andries
Replies (2)
RE: compression netcdf file with cdo version 2.4.0 - Added by Uwe Schulzweida about 1 month ago
Hello Andries,
There are no known compression issues with CDO 2.4.0, and it works fine for me. I suspect that NetCDF was installed without zip support. You can use ‘nc-config --all’ to determine which libraries were linked with NetCDF.
Cheers,
Uwe
RE: compression netcdf file with cdo version 2.4.0 - Added by Andries De Vries about 1 month ago
Hello Uwe,
Thank you so much for your reply. According to our HPC support, there is a different problem. For the moment, we found a solution by using cdo v.2.0.0. I copy-past the writing of the HPC support below; perhaps this could help clarifying whether there may be an issue with v.2.4.0 or perhaps you have suggestions for CDO installations on our cluster?
Thank you very much.
Best,
Andries
- _this is the writing from our HPC support _There is indeed a problem with the zip library but it is not the one indicated by the [cdo] developers. The NetCDF library compiled on the cluster has zip library support. The problem in our case is that we used a different zip library and by changing it to the standard one, the compression has improved.
Now for 1.4G it goes to 1.2.
There is clearly a problem with the version, it is not related to how it is installed in the cluster.
When I tested using Ubuntu packages and the size of the compressed file is:
version size
2.5.0 1.2 G
1.9.3 ~800 M
Mostly the same thing that you observed. I do not think that the version 2.5.0 compiled by Ubuntu/Debian has the same problem as the one on the cluster. We use different pipelines to compile the software.
Fortunately, I was able to install version 2.0.0 which has the same compression performance as the one your testing on the other cluster (version 1.9.6). I used exactly the same method to compile it. So, this is another proof that there is a problem with the version.