Project

General

Profile

Forums » Support »

resmapcon producing gridcells in totally chaotic order

Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

I am trying to transform VPRM flux data from the WRF Lambertian Azimutal Grid to a regular ~5km grid. Here is how the structure of this file looks like:

netcdf vprm_fluxes_EU5km_20150101 {
dimensions:
west_east = 962 ;
south_north = 775 ;
time = 24 ;
grid_corners = 4 ;
grid_rank = 2 ;
variables:
double year(time) ;
double month(time) ;
double day(time) ;
double hour(time) ;
double lat(south_north, west_east) ;
lat:units = "degrees_N" ;
lat:long_name = "Latitude" ;
lat:bounds = "grid_corner_lat" ;
double lon(south_north, west_east) ;
lon:units = "degrees_E" ;
lon:long_name = "Longitude" ;
lon:bounds = "grid_corner_lon" ;
double NEE ;
NEE:units = "umol/m2/s" ;
NEE:long_name = "net ecosystem exchange" ;
NEE:coordinates = "lon lat" ;
double RESP ;
RESP:units = "umol/m2/s" ;
RESP:long_name = "respiration" ;
RESP:coordinates = "lon lat" ;
double GPP ;
GPP:units = "umol/m2/s" ;
GPP:long_name = "gross primary productivity" ;
GPP:coordinates = "lon lat" ;
double grid_corner_lat(grid_corners, south_north, west_east) ;
grid_corner_lat:units = "degrees" ;
grid_corner_lat:_FillValue = NaN ;
double grid_corner_lon(grid_corners, south_north, west_east) ;
grid_corner_lon:units = "degrees" ;
grid_corner_lon:_FillValue = NaN ;
int grid_dims(grid_rank) ;
grid_dims:_FillValue = -2147483648 ;
}

The target grid is in the attachment.

The command is then

cdo -P 8 -remapcon,CHE_STILT_Jena_grid.txt -selname,GPP,NEE,RESP -settunits,hours -settaxis,2015-01-01,00:00:00,1hour vprm_fluxes_EU5km_20150101.nc vprm_fluxes_CHE_EU5_lonlat_20150101.nc

The output file has the following structure:

netcdf vprm_fluxes_CHE_EU5_lonlat_20150101 {
dimensions:
time = UNLIMITED ; // (24 currently)
lon = 1440 ;
bnds = 2 ;
lat = 1008 ;
variables:
double time(time) ;
time:standard_name = "time" ;
time:units = "hours since 2015-1-1 00:00:00" ;
time:calendar = "proleptic_gregorian" ;
time:axis = "T" ;
double lon(lon) ;
lon:standard_name = "longitude" ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
lon:axis = "X" ;
lon:bounds = "lon_bnds" ;
double lon_bnds(lon, bnds) ;
double lat(lat) ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
lat:axis = "Y" ;
lat:bounds = "lat_bnds" ;
double lat_bnds(lat, bnds) ;
double NEE ;
NEE:long_name = "net ecosystem exchange" ;
NEE:units = "umol/m2/s" ;
NEE:_FillValue = -9.e+33 ;
NEE:missing_value = -9.e+33 ;
double RESP ;
RESP:long_name = "respiration" ;
RESP:units = "umol/m2/s" ;
RESP:_FillValue = -9.e+33 ;
RESP:missing_value = -9.e+33 ;
double GPP ;
GPP:long_name = "gross primary productivity" ;
GPP:units = "umol/m2/s" ;
GPP:_FillValue = -9.e+33 ;
GPP:missing_value = -9.e+33 ;

// global attributes:
:CDI = "Climate Data Interface version 1.9.8 (https://mpimet.mpg.de/cdi)" ;
:Conventions = "CF-1.6" ;
:history = "Thu Jan 30 23:56:14 2020: cdo -P 8 -remapcon,CHE_STILT_Jena_grid.txt -selname,GPP,NEE,RESP -settunits,hours -settaxis,2015-01-01,00:00:00,1hour vprm_fluxes_EU5km_20150101.nc vprm_fluxes_CHE_EU5_lonlat_20150101.nc" ;
:CDO = "Climate Data Operators version 1.9.8 (https://mpimet.mpg.de/cdo)" ;
:cdo_openmp_thread_number = 8 ;
}

However, the order of the gridcells seems to have been completely lost. I would appreciate any help.

Best,
Tonatiuh

CHE_STILT_Jena_grid.txt (73 KB) CHE_STILT_Jena_grid.txt

Replies (25)

RE: resmapcon producing gridcells in totally chaotic order - Added by Karin Meier-Fleischer about 4 years ago

Hi Tonatiuh,

without the data it is not clear what you mean and we are not able to help. The grid file seems to be OK.

Do you have had a look at

cdo sinfon vprm_fluxes_CHE_EU5_lonlat_20150101.nc

Compare the grid information of you gridfile and the output file

cdo griddes vprm_fluxes_CHE_EU5_lonlat_20150101.nc

Have you compared the input and output file plots for a single variable.

-Karin

RE: remapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Dear Karin,

The files are very large so I have put them under ftp.zmaw.de/incoming

The input file is vprm_fluxes_EU5km_20150101.nc
The output file is prm_fluxes_CHE_EU5_lonlat_20150101.nc

Here is the cdo sinful comparison:
For the input file I get:

cdo sinfon: Processed 8 variables over 24 timesteps [0.26s 118MB].
File format : NetCDF
-1 : Institut Source T Steptype Levels Num Points Num Dtype : Parameter name
1 : unknown unknown v instant 1 1 1 1 F64 : year
2 : unknown unknown v instant 1 1 1 1 F64 : month
3 : unknown unknown v instant 1 1 1 1 F64 : day
4 : unknown unknown v instant 1 1 1 1 F64 : hour
5 : unknown unknown v instant 1 1 745550 2 F64 : NEE
6 : unknown unknown v instant 1 1 745550 2 F64 : RESP
7 : unknown unknown v instant 1 1 745550 2 F64 : GPP
8 : unknown unknown c instant 1 1 2 3 I32 : grid_dims
Grid coordinates :
1 : generic : points=1
2 : curvilinear : points=745550 (962x775)
lon : -35.55054 to 60.55054 degrees_E
lat : 31.00161 to 68.7148 degrees_N
available : cellbounds
3 : generic : points=2
Vertical coordinates :
1 : surface : levels=1
Time coordinate : 24 steps
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00
0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00
0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00
0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00
0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00
0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00 0000-00-00 00:00:00

For the output file, I get:

cdo sinfon: Processed 3 variables over 24 timesteps [0.78s 34MB].
File format : NetCDF
-1 : Institut Source T Steptype Levels Num Points Num Dtype : Parameter name
1 : unknown unknown v instant 1 1 1451520 1 F64 : NEE
2 : unknown unknown v instant 1 1 1451520 1 F64 : RESP
3 : unknown unknown v instant 1 1 1451520 1 F64 : GPP
Grid coordinates :
1 : lonlat : points=1451520 (1440x1008)
lon : -29.96875 to 59.96875 by 0.0625 degrees_east
lat : 30.02083 to 71.97917 by 0.04166667 degrees_north
available : cellbounds
Vertical coordinates :
1 : surface : levels=1
Time coordinate : 24 steps
RefTime = 2015-01-01 00:00:00 Units = hours Calendar = proleptic_gregorian
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
2015-01-01 00:00:00 2015-01-01 01:00:00 2015-01-01 02:00:00 2015-01-01 03:00:00
2015-01-01 04:00:00 2015-01-01 05:00:00 2015-01-01 06:00:00 2015-01-01 07:00:00
2015-01-01 08:00:00 2015-01-01 09:00:00 2015-01-01 10:00:00 2015-01-01 11:00:00
2015-01-01 12:00:00 2015-01-01 13:00:00 2015-01-01 14:00:00 2015-01-01 15:00:00
2015-01-01 16:00:00 2015-01-01 17:00:00 2015-01-01 18:00:00 2015-01-01 19:00:00
2015-01-01 20:00:00 2015-01-01 21:00:00 2015-01-01 22:00:00 2015-01-01 23:00:00

The grid description is too large to put here but you can generated from the files I unloaded.

Thank you for having a look into this. I am looking forward to receiving your response.

Best,
Tonatiuh

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Hi Karin,

I am not able to put them in outgoing. I get the message "Operation not permitted" I am not at MPI Hamburg. Could you please tell me the path.

Cheers,
Tonatiuh

RE: resmapcon producing gridcells in totally chaotic order - Added by Karin Meier-Fleischer about 4 years ago

Ok, I thought that you are from MPI. You can use https://nextcloud.dkrz.de/s/W9EAAH7QL4wb8N4 to upload the file today.

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Thank you Karin, I have uploaded the files already. I inadvertently uploaded two files nrcs with them. Please just ignore them and delete them.

RE: resmapcon producing gridcells in totally chaotic order - Added by Karin Meier-Fleischer about 4 years ago

The file vprm_fluxes_CHE_EU5_lonlat_20150101.nc is already a lonlat file! You've missed to upload the input file.

You have to compare the grid of the output file with the gridfile grid! The best would be to plot the input and the output file to see what happen to your data.

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Dear Karin,

The file vprm_fluxes_CHE_EU5_lonlat_20150101.nc is the output file, which is in lonlat. The input file is vprm_fluxes_EU5km_20150101.nc, which I was sure I also uploaded. Maybe something happened, however, the link you sent me before is not working anymore. It says::

file not found

The document could not be found on the server.

Maybe the link was deleted or has expired? Could you please send me a new link so that I may uploaded again in case you did not get it? Thank you.

In the attachment is a screenshot of the input and output files as displayed by ncview.

Best,
Tonatiuh

CDO_input_output_WRF.001.png (645 KB) CDO_input_output_WRF.001.png

RE: resmapcon producing gridcells in totally chaotic order - Added by Karin Meier-Fleischer about 4 years ago

You've uploaded the vprm lonlat file and two nrcs files which I have deleted after your request. The link to the cloud was only available until last Saturday. Here is a new link which is valid until tomorrow, please upload the input file.

https://nextcloud.dkrz.de/s/8NkxjeaqtzwS8Xk

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Dear Karin,

I think this is the wrong link. It only has the file to download but not upload.

Cheers,
Tonatiuh

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Dear Karin,

somehow, I have the website won't accept my file. You can download it from here:

https://oc.bgc-jena.mpg.de/index.php/s/zXj3XfOdMlUMxP2

I hope you can find the problem.

Kind regards,
Tonatiuh

RE: resmapcon producing gridcells in totally chaotic order - Added by Karin Meier-Fleischer about 4 years ago

After I tried to download the file several times I have now given up. It is a very, very slow connection.

Please, upload the file to
https://nextcloud.dkrz.de/s/EaxfR6PgfXEWAS3

I've tested the link in private mode and it works good. The link's expiration date is Friday 07 and it's only for uploading.

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Dear Karin,

It seems to have worked now. Please let me know if you can retrieve the file.

Best,
Tonatiuh

RE: resmapcon producing gridcells in totally chaotic order - Added by Karin Meier-Fleischer about 4 years ago

The conservative remapping is not working for regional data. You can use remapbil or remapbic.
Befor you can use remapnn and remapdis you have to set the environment variable REMAP_EXTRAPOLATE to 'off'.

Remapping with remapnn, just for variable RESP and the first timestep:

export REMAP_EXTRAPOLATE='off'
cdo -remapnn,CHE_STILT_Jena_grid.txt -seltimestep,1 -selvar,RESP vprm_fluxes_EU5km_20150101.nc ooo_RESP_t1_r1440x720_remapnn.nc

screen.png (612 KB) screen.png

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Dear Karin,

Thank you very much for your answer. I have a lot of files with daily cycles. However, when I try to extend your solution to:

export REMAP_EXTRAPOLATE='off'
cdo -remapnn,CHE_STILT_Jena_grid.txt -selname,GPP,NEE,RESP -settunits,hours -settaxis,2015-01-01,00:00:00,1hour vprm_fluxes_EU5km_20150101.nc vprm_resp_fluxes_CHE_EU5_lonlat_20150101.nc

The extrapolation still happens creating spurious values where we see white missing values in your image. Is this possible or do I have to do file by file, variable by variable and time step by time step?

Cheers,
Tonatiuh

RE: resmapcon producing gridcells in totally chaotic order - Added by Karin Meier-Fleischer about 4 years ago

Which version of CDO are you using?

cdo -V

Due to the fact that I can remap the data correctly, it is hard to say what is going wrong on your system.

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Hi Karin,

I get this:

CDI library version : 1.9.8
 cgribex library version : 1.9.4
 ecCodes library version : 2.6.0
  NetCDF library version : 4.6.1 of Jul 18 2018 14:28:10 $
    hdf5 library version : 1.8.14 threadsafe
    exse library version : 1.4.1
    FILE library version : 1.8.3

This is the standard in mistral.

Best,
Tonatiuh

RE: resmapcon producing gridcells in totally chaotic order - Added by Karin Meier-Fleischer about 4 years ago

That's weird, I've run it on mistral with cdo-1.9.8, with and without magicsxx, and it remaps the data correct without artifacts.

screen_1.png (421 KB) screen_1.png

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Dear Karin,

This is indeed strange when I give the cdo command directly in the command line there are no artifacts. However, when I put this in a batch script the artifacts return:

#!/bin/csh
#SBATCH --account=mj0143
#SBATCH --output=cdo_generate_grid_to_regular_CHE_EU5.out
#SBATCH --error=cdo_generate_grid_to_regular_CHE_EU5.err
#SBATCH --partition=compute
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=5300
#SBATCH --no-requeue

export REMAP_EXTRAPOLATE='off'
cdo -remapnn,CHE_STILT_Jena_grid.txt -selname,GPP,NEE,RESP -settunits,hours -settaxis,2015-01-01,00:00:00,1hour vprm_fluxes_EU5km_20150101.nc vprm_resp_fluxes_CHE_EU5_lonlat_20150101.nc

This might be a problem as I would need to cycle over 365 files in a directory. Do you have any idea what is happening?

Thank you.

Best regards,
Tonatiuh

Screenshot 2020-02-07 at 11.05.06.pdf (1.4 MB) Screenshot 2020-02-07 at 11.05.06.pdf

RE: resmapcon producing gridcells in totally chaotic order - Added by Karin Meier-Fleischer about 4 years ago

Ok, that points directly to a problem in your batch environemnt. Please, insert the call 'cdo -V' in your batch script. Which modules do you load in your batch script?

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

1) cdo/1.9.8-magicsxx-gcc64 3) pftp/7.5.1.2-krb5 5) netcdf_c/4.3.2-gcc48 7) defaults 9) intel/18.0.4 11) r/3.5.3
2) imagemagick/6.9.1-7-gcc48 4) lftp/4.9.1 6) git/2.25.0 8) jdk/1.8.0_20 10) ncview/2.1.4-gcc48

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

This is what I get once I include cdo -V

Climate Data Operators version 1.9.8 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -fopenmp -O2 -g -march=native -mavx2
CXX version : g++ (GCC) 6.4.0
C Compiler: gcc -fopenmp -pthread -O2 -g -march=native -mavx2
C version : gcc (GCC) 6.4.0
F77 Compiler: gfortran -O2 -g -march=native -mavx2
F77 version : GNU Fortran (GCC) 6.4.0
Features: 251GB 48threads C++14 Fortran DATA PTHREADS OpenMP HDF5 NC4/HDF5/threadsafe OPeNDAP SZ UDUNITS2 PROJ MAGICS CURL FFTW3 CMOR AVX2
Libraries: HDF5/1.8.14 proj/6.1.1 CMOR/3.5.0 curl/7.19.7
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.8
cgribex library version : 1.9.4
ecCodes library version : 2.6.0
NetCDF library version : 4.6.1 of Jul 18 2018 14:28:10 $
hdf5 library version : 1.8.14 threadsafe
exse library version : 1.4.1
FILE library version : 1.8.3

export: Command not found.
cdo(1) selname: Process started
cdo(2) settunits: Process started
cdo(3) settaxis: Process started
Warning (find_time_vars): Time variable >time< not found!
cdo(2) settunits: Changing absolute to relative time axis!
cdo remapnn: Nearest neighbor weights from curvilinear (962x775) to lonlat (1440x1008) grid
cdo(3) settaxis: Processed 8 variables over 24 timesteps.
cdo(2) settunits: Processed 53679698 values from 8 variables over 24 timesteps.
cdo(1) selname: Processed 53679600 values from 8 variables over 24 timesteps.
cdo remapnn: Processed 53679600 values from 3 variables over 24 timesteps [5.79s 395MB].

RE: resmapcon producing gridcells in totally chaotic order - Added by Tonatiuh Guillermo Nuñez Ramirez about 4 years ago

Oh I found the problem I had #!/bin/csh instead of #!/bin/bash. Thank you for pointing me in the right way.

Best regards,
Tonatiuh

    (1-25/25)