Project

General

Profile

Generating ycon remaping weight for a large grid took a long time without progress

Added by Yi Zhang almost 3 years ago

Dear CDO,
I use "cdo -P 32 genycon,${cdo_grid_file} var.tmp.nc grid5.ycon.weight.nc" to generate a remapping file for icosahedral-hexagonal 5km mesh to


Replies (4)

RE: Generating ycon remaping weight for a large grid took a long time without progress - Added by Yi Zhang almost 3 years ago

Sorry for the unfinished message... As in that line, var.tmp.nc is a regular 0.25-degree-resolution file that only contains one variable. The target is an unstructured icosahedral-hexagonal mesh has 23592962 cells (this mesh descriptor for CDO is ~7.7g so I cannot upload). This command works well for lower-resolution cases. But for this large file, it has run for several days without any progress (e.g., percent information). I don't know whether CDO is still behaving properly and doing the remapping, but just needs more time. Or it has been stuck for some reasons.

RE: Generating ycon remaping weight for a large grid took a long time without progress - Added by Yi Zhang almost 3 years ago

The progress is still zero till now. I have uploaded the ${cdo_grid_file} and var.tmp.nc here: https://zenodo.org/record/4925960; if anyone can help with this.

RE: Generating ycon remaping weight for a large grid took a long time without progress - Added by Uwe Schulzweida almost 3 years ago

Thanks for the data, this helps us a lot to analyze the problem!
Your ASCII grid description file g9b3_file_prime_cell.cdo has a size of ~8GB. It turns out that the maximum size in CDO is only 2GB. With larger files there is an integer overflow. This integer overflow leads to an undefined behavior. We will solve this problem in the next CDO version 2.0.0.
In terms of performance it is not a good idea to read large ASCII files. You can also simply specify a NetCDF data file which also contains the grid information. This is much faster and the above error does not occur:

cdo -P 32 genycon,datafile_with_grid.nc var.tmp.nc grid5.ycon.weight.nc

RE: Generating ycon remaping weight for a large grid took a long time without progress - Added by Yi Zhang almost 3 years ago

Hi, Uwe, thanks so much for your help. I can do this remapping now by using a SCRIP-style NetCDF grid description file. Indeed fast!

    (1-4/4)