CDO

Hello Paul,

The NetCDF support in CDO is limited to a subset of the NetCDF CF-convention. Coordinates that do not follow this convention cannot be processed correctly in CDO. For the datetime format of WRF files we have made an exception and converted them to the CF convention. Unfortunately this cannot be converted back to the WRF format, sorry.

Cheers,
Uwe

Thank you. Unfortunately, I don't know any other way to get the format I need. And I really need those SST from previous wrfout* files in my downscaling run.

I tried ncrcat to merge Times variable (only) from another WRF file to the output from CDO, whom I deleted Times variable ; didn't work, not much surprise here.

May I ask :

- Why the "reverse" conversion to CF convention isn't possible/added to the code ? Would it be so complicated to add ?
- Why convert the Times values to CF convention in first place ? Is it necessary for the remapbil operation ?
- Lastly, would you please have something in mind that could help me ?

Thank you very much for your help.

The I/O bibliotek in CDO supports different file formats. For this, an abstract data model is used, which does not correspond exactly to the NetCDF data model. This data model supports only a part of the CF convention. Therefore sometimes some information is lost. Also no specific information from the input is known when writing the data.
I think you need a NetCDF tool like NCO. With this you can also interpolate the data: https://nco.sourceforge.net/nco.html#Regridding
Unfortunately I can't help you with this, because I have never used it.

Thank you for your time. I got some help and was able to delete Times in CDO output files, then rewrite (copy in) the correct Times from the original CDO input. It was a little bit trick with NCO tools but it worked.

For others who may face the same problem with WRF files ( you'll need NCO ) :

#1 Create grid description file
cdo griddes wanted_grid_file.nc > griddes.grd

#2 Remap according to grid descrition file
cdo remapbil,griddes.grd, infile.nc outfile.nc

#3 Renamme variable
ncrename -d Times,Time outfile.nc

#4 Delete uncorrectly formated datetimes in outfile.nc
ncks -x -v Times outfile.nc outfile.timeless.nc

#5 Add correct datetimes format in outfile from another file (infile.nc for example)
ncks -A -v Times correct_datetime_format_file.nc outfile.timeless.nc

#6 Rename outfile.timeless.nc
mv outfile.timeless.nc outfile.nc

Now : outfile.nc has got wanted (WRF) formated datetimes

Uwe Schulzweida wrote in RE: Remapbil changed datetime format from WRF files:

Hello Paul,

The NetCDF support in CDO is limited to a subset of the NetCDF CF-convention. Coordinates that do not follow this convention cannot be processed correctly in CDO. For the datetime format of WRF files we have made an exception and converted them to the CF convention. Unfortunately this cannot be converted back to the WRF format, sorry.

Cheers,
Uwe

Hi Uwe,

Could you confirm the following error has the same cause, i.e. WRF netCDF format ?

pmauger@datarmor2:/home3/datawork/pmauger/CONFIGS/TEST_WRFINPUT_FROM_CYR> cdo griddes wrfinput_d01_2005_01 > d01.txt
Warning (cdf_set_var): Inconsistent variable definition for XLAT_U!
Warning (cdf_set_var): Inconsistent variable definition for XLONG_U!
Warning (cdf_set_var): Inconsistent variable definition for XLAT_V!
Warning (cdf_set_var): Inconsistent variable definition for XLONG_V!
cdo griddes: Processed 124 variables [0.11s 27MB]
pmauger@datarmor2:/home3/datawork/pmauger/CONFIGS/TEST_WRFINPUT_FROM_CYR> cdo remapbil,d01.txt, wrfinput_origin.nc wrfinput_d01_2005-01
Warning (cdf_set_var): Inconsistent variable definition for XLAT_U!
Warning (cdf_set_var): Inconsistent variable definition for XLAT_V!
Warning (cdf_set_var): Inconsistent variable definition for XLONG_U!
Warning (cdf_set_var): Inconsistent variable definition for XLONG_V!
Warning (find_time_vars): Time variable >Time< not found!

cdo remapbil (Abort): Unsupported generic coordinates (Variable: DN)!

Could you please explain why the WRF staggered grid isn't compliant but the mass grid is ?

Thank you

Without some sample data it's impossible to comment on that any further, sry.

Ralf Mueller wrote in RE: Remapbil changed datetime format from WRF files:

Without some sample data it's impossible to comment on that any further, sry.

Hello, I uploaded the original file and thegrid-description file. Thank you for your help understanding the problem, eventually leading to a solution.

The good¶

Let's start with the reasonable part of WRF output: the variables using XLONG/XLAT coordinates/grid

Those variables like CANWAT or COSALPHA or E use the CF-conform linking between the data variable and the real coordinate variables by using the variable attribute coordinates = "XLONG XLAT" which those two variables are present in the file and use a subset of the dimensions (non-timedependent in case of CDO) as the data variables:

        float E(Time, south_north, west_east) ;
                E:FieldType = 104 ;
                E:MemoryOrder = "XY " ;
                E:description = "Coriolis cosine latitude term" ;
                E:units = "s-1" ;
                E:stagger = "" ;
                E:coordinates = "XLONG XLAT" ;
        float XLONG(Time, south_north, west_east) ;
                XLONG:FieldType = 104 ;
                XLONG:MemoryOrder = "XY " ;
                XLONG:description = "LONGITUDE, WEST IS NEGATIVE" ;
                XLONG:units = "degree_east" ;
                XLONG:stagger = "" ;
        float XLAT(Time, south_north, west_east) ;
                XLAT:FieldType = 104 ;
                XLAT:MemoryOrder = "XY " ;
                XLAT:description = "LATITUDE, SOUTH IS NEGATIVE" ;
                XLAT:units = "degree_north" ;
                XLAT:stagger = "" ;

The bad¶

Purely vertical variables are created with non-CF-conform vertical dimensions:

        float RDN(Time, bottom_top) ;
                RDN:FieldType = 104 ;
                RDN:MemoryOrder = "Z  " ;
                RDN:description = "inverse d(eta) values between half (mass) levels" ;
                RDN:units = "" ;
                RDN:stagger = "" ;

bottom_top is nothing but a dimension without any meta data like it is done for XLONG. Hence CDO does the default: it assumes the dimension is a horizontal one. There is nothing else to do. The naming convention for dimensions used in WRF is nice, but totally useless for general tools. My personal consequence is: Dimension names have no meaning - only data variables have (I use terms of the CF-convention here).

The ugly¶

Variables on the velocity grid like U:

        float U(Time, bottom_top, south_north, west_east_stag) ;
                U:FieldType = 104 ;
                U:MemoryOrder = "XYZ" ;
                U:description = "x-wind component" ;
                U:units = "m s-1" ;
                U:stagger = "X" ;
                U:coordinates = "XLONG_U XLAT_U" ;
        float XLONG_U(Time, south_north, west_east_stag) ;
                XLONG_U:FieldType = 104 ;
                XLONG_U:MemoryOrder = "XY " ;
                XLONG_U:description = "LONGITUDE, WEST IS NEGATIVE" ;
                XLONG_U:units = "degree_east" ;
                XLONG_U:stagger = "X" ;
                XLONG_U:coordinates = "XLONG_U XLAT_U" ;
        float XLAT_U(Time, south_north, west_east_stag) ;
                XLAT_U:FieldType = 104 ;
                XLAT_U:MemoryOrder = "XY " ;
                XLAT_U:description = "LATITUDE, SOUTH IS NEGATIVE" ;
                XLAT_U:units = "degree_north" ;
                XLAT_U:stagger = "X" ;
                XLAT_U:coordinates = "XLONG_U XLAT_U" ;

BUT: The coordinate variables XLONG_U/XLAT_U use the coordinates attribute pointing to themselves!

This is simply wrong and will misguide any CF-conform tool. The same error is done for the 3rd grid (XLONG_V/XLAT_V).

I usually check how CDO understand the input by using the sinfov operator.

How to repair the file¶

1. remove the coordinates attribute from the mentioned variables (copy.nc is a copy of your uploaded):

   ncatted -a coordinates,XLONG_U,d,, copy.nc
   ncatted -a coordinates,XLONG_V,d,, copy.nc
   ncatted -a coordinates,XLAT_V,d,, copy.nc
   ncatted -a coordinates,XLAT_U,d,, copy.nc
   

2. add data variables called bottom_top and bottom_top_stag which have an axis attribute set to "Z" and have on the dimension with the appropriate name. Currently I dont know how to create this on the fly. NCO or some python or ruby netcdf bindings will help

Hello,

I've got some help and succeeded at regridding WRF's NetCDF. Please find below the procedure to be used :

1. Split coarse and fine files to T, U and V grids :

T grid :

ncks -v CANWAT,CLAT,COSALPHA,E,F,HGT,ISLTYP,IVGTYP,LAKEMASK,LANDMASK,LU_INDEX,MAPFAC_M,MAPFAC_MX,MAPFAC_MY,MU,MUB,P,PB,PH,PHB,PSFC,Q2,QCLOUD,QGRAUP,QICE,QRAIN,QSNOW,QVAPOR,SEAICE,SH2O,SHDMAX,SHDMIN,SINALPHA,SMCREL,SNOALB,SNOW,SNOWH,SR,SST,T,T2,TH2,TMN,TSK,U10,V10,VAR,VAR_SSO,VEGFRA,W,XLAND,XLAT,XLONG coarse_in.nc coarse_out_T.nc

ncks -v CANWAT,CLAT,COSALPHA,E,F,HGT,ISLTYP,IVGTYP,LAKEMASK,LANDMASK,LU_INDEX,MAPFAC_M,MAPFAC_MX,MAPFAC_MY,MU,MUB,P,PB,PH,PHB,PSFC,Q2,QCLOUD,QGRAUP,QICE,QRAIN,QSNOW,QVAPOR,SEAICE,SH2O,SHDMAX,SHDMIN,SINALPHA,SMCREL,SNOALB,SNOW,SNOWH,SR,SST,T,T2,TH2,TMN,TSK,U10,V10,VAR,VAR_SSO,VEGFRA,W,XLAND,XLAT,XLONG fine_in.nc fine_out_T.nc

U grid :

ncks -v U,MAPFAC_U,MAPFAC_UX,MAPFAC_UY,XLAT_U,XLONG_U coarse_in.nc coarse_out_U.nc

ncks -v MAPFAC_U,MAPFAC_UX,MAPFAC_UY,XLAT_U,XLONG_U fine_in.nc fine_out_U.nc

V grid :

ncks -v V,MAPFAC_V,MAPFAC_VX,MF_VX_INV,MAPFAC_VY,XLAT_V,XLONG_V coarse_in.nc coarse_out_V.nc

ncks -v V,MAPFAC_V,MAPFAC_VX,MF_VX_INV,MAPFAC_VY,XLAT_V,XLONG_V fine_in.nc fine_out_V.nc

2. Remap each of the new coarse grids to fine grids :

cdo griddes fine_in_T.nc > grid_T.txt 
cdo remapbil,grid_T.txt coarse_out_T.nc fine_out_T_remap.nc

cdo griddes fine_in_U.nc > grid_U.txt 
cdo remapbil,grid_U.txt coarse_out_U.nc fine_out_U_remap.nc

cdo griddes fine_in_V.nc > grid_V.txt 
cdo remapbil,grid_V.txt coarse_out_V.nc fine_out_V_remap.nc

3. Concatenate T, U and V remapped fine grids together :

cp fine_out_T_remap.nc fine_out_all_remap.nc
ncks -A fine_out_U_remap.nc fine_out_all_remap.nc
ncks -A fine_out_V_remap.nc fine_out_all_remap.nc

4. Add times :

ncrename -d time,Time fine_out__all_remap.nc
ncks -A -v Times fine_in.nc fine_out_all_remap.nc

thx for posting your solution!

With my suggestions taken into account, splitting the input along the provided grids can be done by

cdo splitgrid <input file>

cheers,
ralf

Update for complete wrfinput/wrfout processing (compliant with WRF3.6.1 : variables list has been updated since)

I had some trouble with my previous script version. Some variables on the W staggered grid that I has separated from the unstaggered grid woudn't be accepted by NCO (PH, PHB, SH2O, W).
I corrected the problem by suppressing the division of wrfinput_ and wrfout_ into T,U,V axes instead of T,U,V,W. Then only SH2O wouldn't be accepted by NCO (layers number problem). This is why, hereunder, I tested each variable one by one the end of the code. I have to optimize this part.

# DEFINE INPUTS

# USUALLY $1 = wrfinput_d0*
# USUALLY $2 = wrfout_*

# SELECT FIRST TIMESTEP FROM WRFOUT

    ncks -O -d Time,0 $2 wrfout

# WRFOUT (1 timestep) ACCORDING TO T,U,V,W

ncks -v Times,MU,MUB,P,P00,PB,PH,PHB,PSFC,P_STRAT,Q2,QCLOUD,QGRAUP,QICE,QRAIN,QSNOW,QVAPOR,SST,T,T00,T2,TH2,TSK,U10,V10,W wrfout wrfout_T
ncks -v Times,U wrfout wrfout_U
ncks -v Times,V wrfout wrfout_V

#  ACCORDING TO T,U,V,W

cp $1 wrfin

ncks -v Times,ALBBCK,CANWAT,CF1,CF2,CF3,CFN,CFN1,CLAT,COSALPHA,DN,DNW,DTBC,DTS,DTSEPS,E,F,FCX,FNDALBSI,FNDICEDEPTH,FNDSNOWH,FNDSNOWSI,FNDSOILW,FNM,FNP,FRC_URB2D,GCX,HGT,ISLTYP,IVGTYP,LAI,LAKE_DEPTH,LAKE_DEPTH_FLAG,LAKEFLAG,LAKEMASK,LANDMASK,LU_INDEX,MAPFAC_M,MAPFAC_MX,MAPFAC_MY,P_HYD,P_TOP,QV_BASE,RDN,RDNW,RDX,RDY,RESM,SAVE_TOPO_FROM_REAL,SEAICE,SHDMAX,SHDMIN,SINALPHA,SNOALB,SNOW,SNOWC,SNOWH,SR,STEP_NUMBER,T_BASE,T_INIT,TISO,TLP,TLP_STRAT,TMN,TOPOSLPX,TOPOSLPY,U_BASE,U_FRAME,UOCE,VAR,VAR_SSO,V_BASE,VEGFRA,V_FRAME,VOCE,XLAND,XLAT,XLONG,Z_BASE,ZETATOP,ZNU,CPLMASK,DZS,LANDUSEF,SMCREL,SMOIS,SOILCBOT,SOILCTOP,TSLB,ZNW,ZS wrfin wrfin_T
ncks -v Times,MAPFAC_U,MAPFAC_UX,MAPFAC_UY,XLAT_U,XLONG_U wrfin wrfin_U
ncks -v Times,MAPFAC_V,MAPFAC_VX,MAPFAC_VY,MF_VX_INV,XLAT_V,XLONG_V wrfin wrfin_V

# GRID DESCRIPTION

cdo griddes wrfin_T > grid_T.txt 
cdo griddes wrfin_U > grid_U.txt 
cdo griddes wrfin_V > grid_V.txt

# REMAP (BILINEAR)

cdo remapbil,grid_T.txt wrfout_T wrfout_T_remap
cdo remapbil,grid_U.txt wrfout_U wrfout_U_remap
cdo remapbil,grid_V.txt wrfout_V wrfout_V_remap

# CONCATENATE ALL SUB-FILES

cp wrfout_T_remap wrfout_remap

ncks -A wrfout_U_remap wrfout_remap
ncks -A wrfout_V_remap wrfout_remap

# REPLACE TIME VARIABLE TYPE AND TIME COORDINATE BY WRFIN

ncrename -d Times,Time wrfout_remap
ncks -O -s 'Times=char(Times)' wrfout_remap wrfout_remap
ncks -O -x -v Times wrfout_remap wrfout_remap
ncks -A -v Times wrfin wrfout_remap

# CHANGE XLAT* & XLONG* ATTRIBUTES

ncks -C -x -v XLONG,XLAT,XLONG_U,XLAT_U,XLONG_V,XLAT_V wrfout_remap wrfout_remap_xlonglat_ok
ncks -A -v XLONG,XLAT,XLONG_U,XLAT_U,XLONG_V,XLAT_V,HGT wrfin wrfout_remap_xlonglat_ok

# APPEND ALL REMAPPED VARIABLES TO WRFINPUT (to be optimized)

#cp wrfin control_0 # FOR TESTS

ncks -A -C -v MU wrfout_remap_xlonglat_ok wrfin 
ncks -A -C -v MUB wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v P wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v P00 wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v PB wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v PSFC wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v P_STRAT wrfout_remap_xlonglat_ok wrfin

#cp wrfin control_P # IDEM

ncks -A -C -v Q2 wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v QCLOUD wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v QGRAUP wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v QICE wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v QRAIN wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v QSNOW wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v QVAPOR wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v SST wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v T wrfout_remap_xlonglat_ok wrfin

#cp wrfin control_SST_T # IDEM

ncks -A -C -v T00 wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v T2 wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v TH2 wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v TSK wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v U10 wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v V10 wrfout_remap_xlonglat_ok wrfin

#cp wrfin control_UV10 # IDEM

ncks -A -C -v U wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v V wrfout_remap_xlonglat_ok wrfin

#cp wrfin control_UV # IDEM

ncks -A -C -v PH wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v PHB wrfout_remap_xlonglat_ok wrfin
ncks -A -C -v W wrfout_remap_xlonglat_ok wrfin

# RENAME PROPERLY ACCORDING TO INPUT FILE

name="$(basename -- $1)" 
cp wrfin $name

# CLEAN

rm -rf grid* wrfout_* wrfin_* *pid* wrfin wrfinput

# TIME COUNTER ENDS

end=$(date +%s)
echo "Elapsed Time: $(($end-$start)) seconds"