Re-gridding on a smaller subset of global data
Added by Chandrakant Singh almost 5 years ago
I have a NetCDF file, let's say only a portion of South America as mentioned in the link below. I want to use a `cdo` operation `remapnn` to remap the NetCDF file from 0.25x0.25 to 0.05x0.05 resolution. For that, I use the code mentioned below, but the results are not what I expect it to be. Can someone provide some clarity to this?
#current size 190x192
cdo remapnn,r950x960 <myold.nc> <mynew.nc>
I always seem to have this problem when remapping a subset of a global map but works fine for the complete global map. It seems like cdo operation assumes that I am always working with a complete global map.
https://stackoverflow.com/questions/59846765/regridding-operations-on-smaller-subsets
Replies (16)
RE: Re-gridding on a smaller subset of global data - Added by Karin Meier-Fleischer almost 5 years ago
Hi Chandrakant,
0.25 x 0.25 degree --> r1440x720 0.05 x 0.05 degree --> r7200x3600
With the remap opertors your data will be interpolated to a global grid.
-Karin
RE: Re-gridding on a smaller subset of global data - Added by Chandrakant Singh almost 5 years ago
Hi Karin, but that's exactly the problem. I do not have global data. I just have data for one subset. And that subset has size 190x192.
RE: Re-gridding on a smaller subset of global data - Added by Karin Meier-Fleischer almost 5 years ago
I've noticed that. Let me explain you what cdo will do with your sub-region data.
And because I do not have your data I'll create an example sub-region file:
cdo -f nc -setrtomiss,-20000,0 -sellonlatbox,-90,-21,-40,15 -topo,r90x45 topo_subregion.nc
cdo sinfon topo_subregion.nc File format : NetCDF2 -1 : Institut Source T Steptype Levels Num Points Num Dtype : Parameter name 1 : unknown unknown c instant 1 1 221 1 F32 : topo Grid coordinates : 1 : lonlat : points=221 (17x13) lon : 272 to 336 by 4 degrees_east lat : -36.81818 to 12.27273 by 4.090909 degrees_north Vertical coordinates : 1 : surface : levels=1 cdo sinfon: Processed 1 variable [0.00s 14MB].
Now, we interpolate this sub-regional data with r1440x720 to a global grid with remapbil.
cdo remapbil,r1440x720 topo_subregion.nc topo_subregion_r1440x720.nc
Let's see what happened:
cdo sinfon topo_subregion_r1440x720.nc File format : NetCDF2 -1 : Institut Source T Steptype Levels Num Points Num Dtype : Parameter name 1 : unknown unknown c instant 1 1 1036800 1 F32 : topo Grid coordinates : 1 : lonlat : points=1036800 (1440x720) lon : 0 to 359.75 by 0.25 degrees_east circular lat : -89.875 to 89.875 by 0.25 degrees_north Vertical coordinates : 1 : surface : levels=1 cdo sinfon: Processed 1 variable [0.00s 14MB].
And here is the plot for the interpolated sub-region:
If you want to use the remapnn operator you have to disable extrapolation.
To do so change the value of the environment variable REMAP_EXTRAPOLATE
export REMAP_EXTRAPOLATE='off' cdo -remapnn,r1440x720 topo_subregion.nc topo_subregion_r1440x720_nn.nc
cdo sinfon topo_subregion_r1440x720_nn.nc File format : NetCDF2 -1 : Institut Source T Steptype Levels Num Points Num Dtype : Parameter name 1 : unknown unknown c instant 1 1 1036800 1 F32 : topo Grid coordinates : 1 : lonlat : points=1036800 (1440x720) lon : 0 to 359.75 by 0.25 degrees_east circular lat : -89.875 to 89.875 by 0.25 degrees_north Vertical coordinates : 1 : surface : levels=1 cdo sinfon: Processed 1 variable [0.00s 14MB].
You can now use sellonlatbox to cut the wanted region from the interpolated data.
RE: Re-gridding on a smaller subset of global data - Added by Chandrakant Singh almost 5 years ago
Hi Karin,
I tried to do what you did. But the results were not what I expected (as above).
>
export REMAP_EXTRAPOLATE='off'
cdo remapnn,r7200x3600 test.nc test2.nc
>
I have now attached data as 'test.nc'. I am not sure why the data is missing.
Can you provide some feedback as to what I am doing wrong?
RE: Re-gridding on a smaller subset of global data - Added by Karin Meier-Fleischer almost 5 years ago
The metadata of your file is not correct
- the file contains NaN values
- the lat and lon variables don't have units
cdo infon test.nc -1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter name 1 : 2000-01-01 00:00:00 0 36480 0 : 0.0000 nan 897.50 : Transpiration 2 : 2000-01-01 00:00:00 0 36480 0 : 0.0000 nan 4372.6 : Transpiration 3 : 2000-01-01 00:00:00 0 36480 0 : 0.0000 nan 0.48484 : Transpiration
ncdump -h test.nc netcdf test { dimensions: time = UNLIMITED ; // (3 currently) lon = 190 ; lat = 192 ; variables: double time(time) ; time:standard_name = "time" ; time:units = "days since 2000-01-01 00:00:00" ; time:calendar = "standard" ; time:axis = "T" ; float lon(lon) ; lon:axis = "X" ; float lat(lat) ; lat:axis = "Y" ; float Transpiration(time, lat, lon) ; Transpiration:units = "mm/year" ; // global attributes: :CDI = "Climate Data Interface version ?? (http://mpimet.mpg.de/cdi)" ; :Conventions = "CF-1.6" ; :history = "Wed Jan 22 14:35:02 2020: cdo mergetime Arie_Nature_CC_Fig2A_Transpiration.nc Arie_Nature_CC_Fig2B_Median_distance.nc Arie_Nature_CC_Fig2C_Tree_contribution.nc test.nc" ; :CDO = "Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/cdo)" ; }
1. Change the NaN to missing value -99999.9
cdo -setmissval,-99999.9 -setmissval,nan test.nc test_missing.nc
2. Add units attribute to lat and lon
ncatted -O -a units,lon,c,c,"degrees_east" \ -a units,lat,a,c,"degrees_north" test_missing.nc
3. Do the remapping
export REMAP_EXTRAPOLATE='off' cdo -remapnn,r1440x720 test_missing.nc test_subregion_r1440x720.nc
Just to mention the time variable has 3 times the same value.
RE: Re-gridding on a smaller subset of global data - Added by Chandrakant Singh almost 5 years ago
Hi Karin,
I followed all the steps.
But step 3 doesn't seem to change anything. I am still missing the values in the top part of South America. I am attaching the test_missing.nc (with operations done until step 2 with the post).
ncdump -h test_subregion_r1440x720.nc
@netcdf test_subregion_r1440x720 {
dimensions:
time = UNLIMITED ; // (1 currently)
lon = 1440 ;
lat = 720 ;
variables:
double time(time) ;
time:standard_name = "time" ;
time:units = "days since 2000-01-01 00:00:00" ;
time:calendar = "standard" ;
time:axis = "T" ;
float lon(lon) ;
lon:standard_name = "longitude" ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
lon:axis = "X" ;
float lat(lat) ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
lat:axis = "Y" ;
float Median_dist(time, lat, lon) ;
Median_dist:units = "km" ;
Median_dist:_FillValue = -99999.9f ;
Median_dist:missing_value = -99999.9f ;
// global attributes:
:CDI = "Climate Data Interface version ?? (http://mpimet.mpg.de/cdi)" ;
:history = "Wed Jan 22 16:10:23 2020: cdo -remapnn,r1440x720 test_missing.nc test_subregion_r1440x720.nc\nWed Jan 22 16:09:54 2020: ncatted -O -a units,lon,c,c,degrees_east -a units,lat,a,c,degrees_north test_missing.nc\nWed Jan 22 16:07:39 2020: ncatted -O -a units,lon,c,c,degrees_east -a units,lat,a,c,degrees_north test_missing.nc\nWed Jan 22 16:07:25 2020: cdo -setmissval,-99999.9 -setmissval,nan Arie_Nature_CC_Fig2B_Median_distance.nc test_missing.nc" ;
:Conventions = "CF-1.6" ;
:NCO = "4.7.2" ;
:CDO = "Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/cdo)" ;@
test_missing.nc (162 KB) test_missing.nc |
RE: Re-gridding on a smaller subset of global data - Added by Karin Meier-Fleischer almost 5 years ago
This seems to be a problem of your visualization tool. I use panoply to create the following plot of your uploaded data.
RE: Re-gridding on a smaller subset of global data - Added by Chandrakant Singh almost 5 years ago
Oh sorry for not being clear enough. The data file attached previously had only the operations done until step 2 (which is 'test_missing.nc').
If I attach the file for step3 (which is 'test_subregion_r1440x720.nc'), it would look something like below figure.
P.S.: I am also using panoply for visualization.
RE: Re-gridding on a smaller subset of global data - Added by Chandrakant Singh almost 5 years ago
Respective file for that is attached.
RE: Re-gridding on a smaller subset of global data - Added by Karin Meier-Fleischer almost 5 years ago
Hm, maybe you should update CDO to the current version 1.9.8.
I've uploaded my test_subregion_r1440x720.nc file.
test_subregion_r1440x720.nc (11.9 MB) test_subregion_r1440x720.nc | Create with cdo 1.9.8 |
RE: Re-gridding on a smaller subset of global data - Added by Chandrakant Singh almost 5 years ago
Hi Karin,
Thanks for the clearing things up.
I just wanted to check one last thing that the figure that you posted above is from 'remapnn' or 'remapbil'?
Because when I try to use remapbil, I get the same results as you posted above. But if I do 'remapnn', then there is data missing as in my original post.
All the analysis were performed cdo version 1.9.3.
Is this a bug or a processing error?
RE: Re-gridding on a smaller subset of global data - Added by Karin Meier-Fleischer almost 5 years ago
The file tests_subregion_r1440x720.nc was computed with remapnn as described above and I used
cdo 1.9.8 and Panoply 4.10.12. I can't say if it is a bug because I don't have cdo 1.9.3 installed.
RE: Re-gridding on a smaller subset of global data - Added by Sagar Parajuli about 1 year ago
I am having this same exact problem. I followed the above instructions provided by Karin but I don't get the expected results. As Chandrakanta noted, I have experienced this problem many times earlier as well. My problem is this:
I have a netcdf file LAI_2020_all_lambert.nc as attached in lambert projection with 411 lon and 361 lat grids, lat range from 32.4905 to 34.1118 degree, and lon range from -116.832 to -114.984 over California. It has the variable named Band1, which has 3d dimension (time, lat, lon). I need to regrid the variable Band1 into 162 lon and 171 lat grids for the same lat and lon range. I am basically trying to regrid at a coarser resolution to match my WRF model grids because I want to replace my WRF model variable (LAI) with this new data. This is what I have done:
cdo -setmissval,-99999.9 -setmissval,nan LAI_2020_all_lambert.nc LAI_2020_all_lambert_new.nc
ncatted -O -a units,lon,c,c,"degrees_east" -a units,lat,a,c,"degrees_north" LAI_2020_all_lambert_new.nc
export REMAP_EXTRAPOLATE='off'
cdo remapbil,r162x171 LAI_2020_all_lambert_new.nc LAI_2020_all_lambert_new_regrid.nc
But the resulting file LAI_2020_all_lambert_new_regrid.nc is kind of empty, I don't see any data in the desired place.
Please help.
RE: Re-gridding on a smaller subset of global data - Added by Estanislao Gavilan about 1 year ago
Hi Sagar,
for you case, I am not sure there is a grid called r162x171 (I tried and I got an empty matrix). The best approach is to define your grid.txt file like this one. Note that the grid size is xsize*ysize
gridtype = lonlat
gridsize = 27702
xname = lon
xlongname = longitude
xunits = degrees_east
yname = lat
ylongname = latitude
yunits = degrees_north
xsize = 171
ysize = 162
xfirst = -116.832
xlast = -114.984
yfirst = 32.4905
ylast = 34.1118
then you do your things
export REMAP_EXTRAPOLATE='off'
cdo remapbil,grid.txt -setmissval,nan LAI_2020_all_lambert.nc LAI_2020_all_lambert_remap.nc
RE: Re-gridding on a smaller subset of global data - Added by Sagar Parajuli about 1 year ago
Hi Estanislao, thank you I greatly appreciate your suggestion. I followed your suggestion, in fact I had done something similar before but I am having a strange problem. Just a note, in the above text file I believe xsize should be 162 and ysize should be 171, which you inadvertently wrote the opposite. Anyway, below is what I used for my target grid file mygrid_d03:
gridtype = lonlat
gridsize = 27702
xsize = 162
ysize = 171
xname = west_east
xunits = "degrees latitude"
yname = south_north
yunits = "degrees longitude"
xfirst = -116.832
xlast = -114.984
xinc = 0.01
yfirst = 32.4905
ylast = 34.1118
yinc = 0.01
Then I regridded the data with the below command as you suggested.
cdo remapbil,mygrid_d03 LAI_2020_all_lambert.nc LAI_2020_all_lambert_regrid.nc
Regridding works just fine but the result is not what I had expected. Somehow, the remapping is shifting the data southeast by some extent. I have attached the figure comparing the output of the original data (LAI_2020_all_lambert.nc) and the LAI_2020_all_lambert_regrid.nc. Obviously when I replace my model input data with this new data generated, I can clearly see the shifting of the data which is causing problems in my model. As you can see, a large portion of the southeast corner is cut out in the regridded file due to the strange shift. I have spent several days already on it but can't understand why this is happening. I can shift the data to overlap with my old data by changing the xfirst and yfirst value arbitrarily but I certainly don't want to do that. Could you please help me figure this out? I appreciate your quick reply.
RE: Re-gridding on a smaller subset of global data - Added by Sagar Parajuli about 1 year ago
My above problem has been solved below, thanks to Estanislao:
https://code.mpimet.mpg.de/boards/1/topics/15311?r=15313#message-15313