Project

General

Profile

remapcon failing?

Added by Brendan DeTracey over 3 years ago

With all the help I have graciously received, I finally reached my destination, but with a couple bumps. I have four polygons I have encoded as SCRIP unstructured grids to be used with remapcon. Two of them work, and two of them fail.

$ cdo remapcon,cdo_scrip_grid_gom.nc -reducegrid,zos_Omon_ACCESS-CM2_historical_r1i1p1f1_mask_185008.nc [ zos_Omon_ACCESS-CM2_historical_r1i1p1f1_gn_185008.nc ] test.nc
cdo(1) reducegrid: Process started
cdo    remapcon: YAC first order conservative weights from unstructured (69782) to unstructured (1) grid
cdo(1) reducegrid: Processed 108000 values from 1 variable over 1 timestep.
cdo    remapcon: Processed 69782 values from 1 variable over 1 timestep [0.25s 79MB].

(cmip6) [brendan@hecla:remapcon_demo]$ cdo remapcon,cdo_scrip_grid_wss.nc -reducegrid,zos_Omon_ACCESS-CM2_historical_r1i1p1f1_mask_185008.nc [ zos_Omon_ACCESS-CM2_historical_r1i1p1f1_gn_185008.nc ] test.nc
cdo(1) reducegrid: Process started
cdo    remapcon: YAC first order conservative weights from unstructured (69782) to unstructured (1) grid
cdo(1) reducegrid: Processed 108000 values from 1 variable over 1 timestep.
cdo    remapcon: Processed 69782 values from 1 variable over 1 timestep [0.23s 79MB].

(cmip6) [brendan@hecla:remapcon_demo]$ cdo remapcon,cdo_scrip_grid_css.nc -reducegrid,zos_Omon_ACCESS-CM2_historical_r1i1p1f1_mask_185008.nc [ zos_Omon_ACCESS-CM2_historical_r1i1p1f1_gn_185008.nc ] test.nc
cdo(1) reducegrid: Process started
cdo    remapcon: YAC first order conservative weights from unstructured (69782) to unstructured (1) grid
ERROR: invalid cell

Aborting in file clipping.c, line 1305 ...

(cmip6) [brendan@hecla:remapcon_demo]$ cdo remapcon,cdo_scrip_grid_ess.nc -reducegrid,zos_Omon_ACCESS-CM2_historical_r1i1p1f1_mask_185008.nc [ zos_Omon_ACCESS-CM2_historical_r1i1p1f1_gn_185008.nc ] test.nc
cdo(1) reducegrid: Process started
cdo    remapcon: YAC first order conservative weights from unstructured (69782) to unstructured (1) grid
ERROR: invalid cell

Aborting in file clipping.c, line 1305 ...

All four unstructured grids were produced with the exact same code. Corner points are counter-clockwise. The plot below shows all four polygons. The black dots are the source grid corners. The red dots are the source grid centers of unmasked cells.

Why does the remap fail for two of the target grids? There are viable source grid cells that intersect.


Replies (15)

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

Ah. I see that the relevant code is YAC not cdo. :(

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

I compiled cdo setting #define YAC_VERBOSE_CLIPPING in src/lib/yac/clipping.c . I hoped it would help me debug but no extra information was printed.

I noticed by mistake that an older cdo (1.9.3) worked:

$ cdo -V
Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/cdo)
CXX Compiler: g++ -g -O2 -fdebug-prefix-map=/build/cdo-g3Qjnd/cdo-1.9.3+dfsg.1=. -fstack-protector-strong -Wformat -Werror=format-security -fopenmp
CXX version : g++ (Ubuntu 7.3.0-1ubuntu1) 7.3.0
C Compiler: gcc -g -O2 -fdebug-prefix-map=/build/cdo-g3Qjnd/cdo-1.9.3+dfsg.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -pedantic -fPIC -fopenmp
C version : gcc (Ubuntu 7.3.0-1ubuntu1) 7.3.0
F77 Compiler: f77 -g -O2 -fdebug-prefix-map=/build/cdo-g3Qjnd/cdo-1.9.3+dfsg.1=. -fstack-protector-strong
F77 version : unknown
Features: 62GB C++14 Fortran DATA PTHREADS OpenMP45 HDF5 NC4/HDF5/threadsafe OPeNDAP SZ UDUNITS2 PROJ.4 MAGICS CURL FFTW3 SSE2
Libraries: HDF5/1.10.0 proj/4.93 curl/7.58.0
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
     CDI library version : 1.9.3
GRIB_API library version : 2.6.0
  NetCDF library version : 4.6.0 of Feb  9 2018 19:21:24 $
    HDF5 library version : library undefined
    EXSE library version : 1.4.0
    FILE library version : 1.8.3

$ cdo  reducegrid,zos_Omon_ACCESS-CM2_historical_r1i1p1f1_mask_185008.nc zos_Omon_ACCESS-CM2_historical_r1i1p1f1_gn_185008.nc reduce.nc
cdo reducegrid: Processed 108000 values from 1 variable over 1 timestep [0.08s 85MB]
$ cdo remapcon,cdo_scrip_grid_ess.nc reduce.nc test.nc
cdo remapcon: SCRIP first order conservative weights from unstructured (69782) to unstructured (1) grid
cdo remapcon: Map weight > 1! grid1idx=55259 grid2idx=0 nlink=0 wts=1.13276
cdo remapcon: Map weight > 1! grid1idx=55430 grid2idx=0 nlink=3 wts=1.14521
cdo remapcon: Map weight > 1! grid1idx=55431 grid2idx=0 nlink=4 wts=1.153
cdo remapcon: Map weight < 0! grid1idx=55600 grid2idx=0 nlink=6 wts=-0.154891
cdo remapcon: Map weight < 0! grid1idx=55762 grid2idx=0 nlink=7 wts=-0.367269
cdo remapcon: Map weight < 0! grid1idx=55761 grid2idx=0 nlink=8 wts=-0.832767
cdo remapcon: Map weight < 0! grid1idx=55923 grid2idx=0 nlink=9 wts=-0.358429
cdo remapcon: Map weight < 0! grid1idx=55922 grid2idx=0 nlink=10 wts=-0.960929
cdo remapcon: Map weight < 0! grid1idx=55760 grid2idx=0 nlink=11 wts=-0.236372
cdo remapcon: Map weight < 0! grid1idx=55596 grid2idx=0 nlink=12 wts=-1.16865
cdo remapcon: Processed 69782 values from 1 variable over 1 timestep [0.17s 66MB]

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

Looks like my target polygons were too complicated. I simplified them and the problem went away.

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

I spoke too soon. YAC still fails with "invalid cell" on some very simply polygons. The relevant function in YAC clipping.c is:

static int get_cell_points_ordering(struct point_list * cell) {

  if ((cell->start == NULL) || (cell->start == cell->start->next))
    yac_internal_abort_message("ERROR: invalid cell\n", __FILE__, __LINE__);

  double norm_vec[3];
  struct point_list_element * curr = cell->start;

  compute_norm_vector(curr->vec_coords, curr->next->vec_coords, norm_vec);

  curr = curr->next;

  if (curr->next == cell->start)
    yac_internal_abort_message("ERROR: invalid cell\n", __FILE__, __LINE__);

  do {
    curr = curr->next;

    double dot = dotproduct(curr->vec_coords, norm_vec);

    if (fabs(dot) > tol)
      return dot > 0;

  } while (curr != cell->start);

  return -1;
}

Any advice on what this means and/or how to follow this issue up with YAC maintainers?

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

I forgot to include the simplified polygon and its ncdump:

]$ ncdump cdo_scrip_grid_ess.nc
netcdf cdo_scrip_grid_ess {
dimensions:
        grid_size = 1 ;
        grid_corners = 6 ;
        grid_rank = 1 ;
variables:
        int grid_dims(grid_rank) ;
        double grid_center_lat(grid_size) ;
                grid_center_lat:units = "degrees" ;
        double grid_center_lon(grid_size) ;
                grid_center_lon:units = "degrees" ;
        double grid_corner_lat(grid_size, grid_corners) ;
                grid_corner_lat:units = "degrees" ;
        double grid_corner_lon(grid_size, grid_corners) ;
                grid_corner_lon:units = "degrees" ;
        int grid_imask(grid_size) ;
                grid_imask:units = "unitless" ;

// global attributes:
                :title = "cdo_scrip_grid_ess.nc" ;
data:

 grid_dims = 1 ;

 grid_center_lat = 44.5752793657899 ;

 grid_center_lon = -59.8509057269732 ;

 grid_corner_lat =
  44.9239310821268, 42.9987809907331, 43.7583337517241, 44.3104173896552,
    43.9447807410383, 44.5916675448276 ;

 grid_corner_lon =
  -62.0002590886872, -62.0003759902259, -59.1340888451411, -59.3749980689655,
    -58.8416644413793, -57.1945735855501 ;

 grid_imask = 1 ;
}

cdo_scrip_grid_ess.nc (684 Bytes) cdo_scrip_grid_ess.nc Simplified version

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

clipping.c is failing on the target grid in this part of the code:

  curr = curr->next;

  if (curr->next == cell->start)
    yac_internal_abort_message("ERROR: invalid cell\n", __FILE__, __LINE__);

I do not understand why the pointers are the same.

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

Error originates in a call to yac_cell_clipping in cdo_compute_concave_overlap_areas in remap_conserv.cc

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

Somebody please shoot me... or take mercy on me and delete this thread...
I screwed up a path to my target grids. It was reusing the old ones which led me on this merry chase today.

RE: remapcon failing? - Added by Ralf Mueller over 3 years ago

hi Brendan! no problem, I can delete whatever you want here. But take a little pity on yourself - you cannot imaging how often I tricked myself with mistakes by this.

IMO the only way to make this beneficial is to think about, how to prevent it in the future. Any have some coffee in the meantime

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

Thanks Ralf. My darth mater(pun intended) thanks you too...
The internet, where mistakes live forever, but only if one bothers to search for them. A metaphor for life.

RE: remapcon failing? - Added by Ralf Mueller over 3 years ago

alright - then for your peace and violation of the 3rd law of thermodynamics I will reduce entropy by removing this thread later today

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

Interesting. I was suggesting that the thread not be deleted...
Why should bots have all the fun?

RE: remapcon failing? - Added by Ralf Mueller over 3 years ago

you are so damn right.

So whats your conclusion of 2020: Alt-F4 or Ctrl-Alt-Del ?

RE: remapcon failing? - Added by Brendan DeTracey over 3 years ago

F5 (windows only)

    (1-15/15)