CDO

Moin Tobias!

In case you work on levante, you can also post a path here so that I can take a closer look.

The NaNf locations might be there for good reason - I don't know. But if you want to perform any remapping your have to use the value of the _FillValue or missing_value instead of NaN. Locations with missing values are excluded from the interpolation.

Is the type of fract_upland_ctimean float or double? you might consider converting it to double before. Another thing is the compression: You seem to use compressed data since you use -f nc4c -z zip. I cannot recommend that for anything else but archiving. It does slow down the while analysis by a large factor. just saying ;-)

cheers
ralf

Moin Ralf,

I (try to) do the remapping on breeze, but I copied the data file and the weight file to levante at /work/bm1236/m214103/Data/JSB_BCIC. It would be great if you could look into this and give some more advice. Thanks already for the hint to not use -z zip as default, that is good to know.

Cheers Tobias

indeed your data variable is single precision. this might lead to inconsistencies. I checked ga2_upland_fraction_mean_newmiss.nc:

cdo -infov ga2_upland_fraction_mean_newmiss.nc                                                                                     [Mon 2022-10-24|10:02:16]
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 0000-00-00 00:00:00       0 2953929600 2117665960 :      0.0000     0.55094      1.0000 : fract_upland_ctimean

I check the interpolation on levante myself ...

The amount of missing data is find. It is a global field, but only contains values for the land surface. Would I have to convert the data into double precision prior to the remapping? I already tried

cdo -b F64 remap,/pool/data/ICON/grids/public/mpim/0043/icon_grid_0043_R02B04_G.nc,weights.nc inputfile.nc remapped.nc

when I try this my command is killed - no idea why. I try it now on a 512GB main memory knode with

cdo -v -P 128 -f nc4 -gencon,/pool/data/ICON/grids/public/mpim/0043/icon_grid_0043_R02B04_G.nc:1 /work/bm1236/m214103/Data/JSB_BCIC/ga2_upland_fraction_mean_newmiss.nc Weights.nc

Ralf Mueller wrote in RE: remaping fails due to numeric conversion not represen...:

when I try this my command is killed - no idea why. I try it now on a 512GB main memory knode with [...]

The 512GB knode is running out of memory I believe

I did it on breeze4 which is dedicated to I/O for large fields and in the summary, the cdo process wrote it used about 475GB of RAM to generate the remap weights. Thus, I had hoped this also would be sufficient for the actual remapping. Does it help, if I try to let your command run on breeze (takes a few hours though)?

I keep trying this with less OpenMP threads, takes about an hour I think.

Anyway, you could split the input and run something like northern and southern hemisphere separately. Another option is to remove the points with missing values and by that reduce in the input grid size. 70% of the whole search is pointless

Finally I could run

/sw/spack-levante/cdo-2.0.6-jkuh4i/bin/cdo -v -P 4 -f nc4 -gencon,/pool/data/ICON/grids/public/mpim/0043/icon_grid_0043_R02B04_G.nc:1 /work/bm1236/m214103/Data/JSB_BCIC/ga2_upland_fraction_mean_newmiss.nc Weights.nc

This looks like an error in the weights file. For the large input field the indices of the grids are stored in the data type uint64. There seems to be a problem reading in these indices. If you don't have too much data you can just interpolate them directly without storing the weights:

breeze: cdo -P 16 remapycon,/pool/data/ICON/grids/public/mpim/0043/icon_grid_0043_R02B04_G.nc inputfile.nc remapped.nc

thx Uwe! I would have had a hard time figuring this out ;-)

cheers
ralf

Yes, this works fine! Thanks for your help, both of you!