Project

General

Profile

Forums » Support »

Using genbil weights for remapbil

Added by Michael Dorfman over 3 years ago

I have the following netcdf which should follow CF-1.7 conventions;

```
netcdf hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_00148_new {
dimensions:
X = 6844 ;
Y = 3980 ;
variables:
int mercator ;
mercator:false_easting = 0.f ;
mercator:false_northing = 0.f ;
mercator:grid_mapping_name = "mercator" ;
mercator:longitude_of_projection_origin = 0.f ;
mercator:standard_parallel = 0.f ;
double X(X) ;
X:long_name = "x coordinate of projection" ;
X:standard_name = "projection_x_coordinate" ;
X:units = "m" ;
double Y(Y) ;
Y:long_name = "y coordinate of projection" ;
Y:standard_name = "projection_y_coordinate" ;
Y:units = "m" ;
ubyte dual_pol_enabled(Y, X) ;
dual_pol_enabled:_FillValue = 255UB ;
dual_pol_enabled:long_name = "dual pol enabled flag" ;
dual_pol_enabled:coordinates = "latitude longitude" ;
double latitude(Y, X) ;
latitude:standard_name = "latitude" ;
latitude:units = "degrees_north" ;
double longitude(Y, X) ;
longitude:standard_name = "longitude" ;
longitude:units = "degrees_east" ;
double hailsize(Y, X) ;
hailsize:_FillValue = -9999. ;
hailsize:long_name = "hailsize" ;
hailsize:grid_mapping = "mercator" ;
hailsize:units = "mm" ;
hailsize:coordinates = "latitude longitude" ;

// global attributes:
:agg_file_count = 148 ;
:Conventions = "CF-1.7" ;
}
```

And the following grid file I want to remap to;
```
netcdf mask_conus {
dimensions:
latitude = 3001 ;
longitude = 6201 ;
variables:
float latitude(latitude) ;
latitude:units = "degrees north" ;
float longitude(longitude) ;
longitude:units = "degrees east" ;
float CONUS ;
}
```

I can run remapbil without any issues with the following command;
```cdo remapbil,mask_conus.nc hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_00148_new.nc out_remapped_new.nc```

However it - expectedly - takes several minutes to run. My hope was to speed it up by generating a weights file first. I used genbil to do this;

```cdo genbil,mask_conus.nc -selname,hailsize hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_00148_new.nc weights.nc```

Which generated the following (massive) weights.nc file;
```
netcdf weights {
dimensions:
src_grid_size = 27239120 ;
dst_grid_size = 18609201 ;
src_grid_rank = 2 ;
dst_grid_rank = 2 ;
num_links = 3734036 ;
num_wgts = 1 ;
variables:
int src_grid_dims(src_grid_rank) ;
int dst_grid_dims(dst_grid_rank) ;
double src_grid_center_lat(src_grid_size) ;
src_grid_center_lat:units = "radians" ;
double dst_grid_center_lat(dst_grid_size) ;
dst_grid_center_lat:units = "radians" ;
double src_grid_center_lon(src_grid_size) ;
src_grid_center_lon:units = "radians" ;
double dst_grid_center_lon(dst_grid_size) ;
dst_grid_center_lon:units = "radians" ;
int src_grid_imask(src_grid_size) ;
src_grid_imask:units = "unitless" ;
int dst_grid_imask(dst_grid_size) ;
dst_grid_imask:units = "unitless" ;
double src_grid_frac(src_grid_size) ;
src_grid_frac:units = "unitless" ;
double dst_grid_frac(dst_grid_size) ;
dst_grid_frac:units = "unitless" ;
int src_address(num_links) ;
int dst_address(num_links) ;
double remap_matrix(num_links, num_wgts) ;

// global attributes:
:title = "SCRIP remapping with CDO" ;
:normalization = "none" ;
:map_method = "Bilinear remapping" ;
:conventions = "SCRIP" ;
:source_grid = "curvilinear" ;
:dest_grid = "lonlat" ;
:history = "30 Oct 2020 : cdo genbil,mask_conus.nc -selname,hailsize hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_00148_new.nc weights.nc" ;
:CDO = "Climate Data Operators version 1.9.9rc1 (https://mpimet.mpg.de/cdo)" ;
}
```

I then try to run remapbil with the newly-generated weights file;

```
cdo remapbil,weights.nc hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_00148_new.nc out_remapped_new.nc
```

But get the following error;

```
OpenMP: num_procs=12 max_threads=1
remapbil,weights.nc hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_00148_new.nc out_remapped_new.nc
cdo remapbil: Point search radius = 180 deg
cdo remapbil: Extrapolation disabled!

cdo remapbil (Abort): Unsupported target grid type (generic)!
```

Am I doing this right? Am I specifying the weights.nc file correctly in my remapbil command? Any thoughts as to why I'm seeing the unsupported target grid type error message?


Replies (8)

RE: Using genbil weights for remapbil - Added by Karin Meier-Fleischer over 3 years ago

Hi Michael,

when you have created the weights file with genbil you have to use remap for remapping, not remapbil.

https://code.mpimet.mpg.de/projects/cdo/embedded/cdo.pdf#subsection.2.12.1

cdo genbil,mask_conus.nc -selname,hailsize hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_0014_new.nc weights.nc

cdo remap,mask_conus.nc,weights.nc hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_0014_new.nc out_remapped_new.nc

-Karin

RE: Using genbil weights for remapbil - Added by Michael Dorfman over 3 years ago

Thanks, Karin! Just needed a push in the right direction :)

One question - I see an execution time decrease when pre-generating the weights, but not a huge impact. It looks like I'm seeing a ~50% speedup with the pre-generated weights. Should I expect more than this? Is there anything I can do to speed this up? I tried deflating the weights.nc - but had the same result when running remap.

(base) mdorfman@LVHFFW3Z2:~$ cdo genbil,mask_conus.nc -selname,hailsize hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_00148_new.nc weights.nc
cdo(1) selname: Process started
cdo    genbil: Bilinear weights from curvilinear (6844x3980) to lonlat (6201x3001) grid, with source mask (1942768)
cdo(1) selname: Processed 2 variables over 1 timestep.
cdo    genbil: Processed 27239120 values from 1 variable over 1 timestep [*27.87s* 4724MB].


(base) mdorfman@LVHFFW3Z2:~$ cdo remap,mask_conus.nc,weights.nc hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_00148_new.nc out_remapped_new.nc
cdo    remap: Bilinear weights from curvilinear (6844x3980) to lonlat (6201x3001) grid, with source mask (24128)
cdo    remap: Processed 2 variables over 1 timestep [*24.88s* 5827MB].


(base) mdorfman@LVHFFW3Z2:~$ cdo remapbil,mask_conus.nc hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_00148_new.nc out_remapped_new.nc
cdo    remapbil: Bilinear weights from curvilinear (6844x3980) to lonlat (6201x3001) grid, with source mask (24128)
cdo    remapbil: Bilinear weights from curvilinear (6844x3980) to lonlat (6201x3001) grid, with source mask (1942768)
cdo    remapbil: Processed 2 variables over 1 timestep [*47.46s* 6682MB].

Thanks,
Mike

RE: Using genbil weights for remapbil - Added by Uwe Schulzweida over 3 years ago

Your dataset contains 2 variables with a different number of missing values:

cdo infon hailsize_conus_2020-10-28T06_00_00Z_to_2020-10-29T06_00_00Z_0014_new.nc 
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 0000-00-00 00:00:00       0 27239120 27214992 :      1.0000      1.0000      1.0000 : dual_pol_enabled
     2 : 0000-00-00 00:00:00       0 27239120 25296352 :     0.57062      49.532      4492.0 : hailsize
For both variables the remapping weights must be calculated.
With genbil only the weights for the first variable are calculated. remap then uses these weights for the first variable and calculates new weights for the second variable.

RE: Using genbil weights for remapbil - Added by Michael Dorfman over 3 years ago

Perfect - we're down to 3 seconds of processing time! Thanks so much for your help!

RE: Using genbil weights for remapbil - Added by Michael Dorfman over 3 years ago

Ah - sorry, one last question - if I try to regrid a different input with identical variable/attribute names but different data with the same weights file, it looks like it then regenerates the weights file when running the remapping. This is what I'm seeing;

cdo genbil,mask_conus.nc -selname,hailsize input.nc weights.nc
works;
cdo remap,mask_conus.nc,weights.nc -selname,hailsize input.nc out_remapped.nc
Rebuilds the weights file;
cdo remap,mask_conus.nc,weights.nc -selname,hailsize similar_input.nc out_remapped.nc

Is this expected? Is there any way to have a single weights file for any input that has the same set of X/Y coordinates and lat/lon values? I've attached the files used above for reference (note, the inputs are slightly different from before).

Download all files
input.nc (2.66 MB) input.nc
similar_input.nc (3.18 MB) similar_input.nc
mask_conus.nc (120 KB) mask_conus.nc

RE: Using genbil weights for remapbil - Added by Michael Dorfman over 3 years ago

Update - I've discovered here (https://code.mpimet.mpg.de/boards/1/topics/5275) it's an issue with the fill value mask being different from file to file. The suggested solution in that thread is to set the fill value to a different value (eg "0"), then create the weights grid from that. Is that still the best practice? I was excited about the very-small weights file, but that was probably because most of the grid was masked.

RE: Using genbil weights for remapbil - Added by Uwe Schulzweida over 3 years ago

If it is possible to find a suitable value for the fill value, then it is the best solution.

    (1-8/8)