CDO operators and problems in reading GRIB1 sub hourly records generated by UPPV2.0 from WRF outputs

Added by Giaiotti Dario almost 10 years ago

Problem
Performing data extraction and post processing from WRF (http://www.wrf-model.org/index.php) model outputs by means of UPP2.0 (http://www.dtcenter.org/wrf-nmm/users/overview/upp_overview.php) package I had some problems in using CDO (https://code.zmaw.de/projects/cdo) utilities on the output GRIB1 files.

I had extracted sub hourly fields, in particular sampling WRF outputs every 10 minutes, generating GRIB1 files through UPP2.0 and then I used such files as input for CDO utilities. The problem is that CDO does not recognize the code 14 as Indicator of unit of time range in the GRIB1 files.

In fact UPP writes the GRIB1 record according to NCEP GRIB1 table 4 (PDS Octet 18) ON388 (http://www.nco.ncep.noaa.gov/pmb/docs/on388/table4.html) for octet 18th in section 1, which differs from standard WMO GRIB1 table 4 (PDS Octet 18) (http://www.wmo.int/pages/prog/www/WDM/Guides/Guide-binary-2.html).

I guess CDO utilities read GRIB1 records using WMO standards, so code 14, which means half an hour in NCEP table 4, is not allowed and the utility fails; the same should be true for the quarter of hour flag (13), but I have not tried that.

An example of CDO error message is reported here below, when a file storing six 10 minutes records is processed by some CDO utility:

Time axis : unlimited steps
RefTime = 2013-09-06 00:00:00 Units = hours Calendar = PROLEPTIC
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
2013-09-06 00:00:00 2013-09-06 00:10:00 2013-09-06 00:20:00
gribDateTime : Time unit 14 unsupported
cgribexGetTimeUnit : GRIB time unit 14 unsupported!
2013-09-06 00:00:00
2013-09-06 00:40:00 2013-09-06 00:50:00
cdo sinfon: Processed 34 variables over 6 timesteps. ( 0.02s )

Solution
I modified the GRIBIT (GRIBIT.F) subroutine in the UPP2.0 suite (UPPV2.0/src/unipost/) avoiding the sub hourly code attribution. At the bottom of this post there is the part of GRIBIT.F replaced.

Still remains the problem CDOs do not recognize a single sub hourly record alone for some parameters; but they do when its is concatenated with other records containing at the beginning a full hour record (i.e. 00 ninute).

Here are some examples or errors got in the standard output.

_cdo showtime GRIB/CRMA_WRF-2-TESI_d03_2013090600+000001.10h.GRIB
Warning (cgribexScanTimestep2) : Param=229.2 level=10 not defined at timestep 1!
cdo showtime: Open failed on >GRIB/CRMA_WRF-2-TESI_d03_2013090600+000001.10h.GRIB<
Unsupported file structure

cdo showtime GRIB/CRMA_WRF-2-TESI_d03_2013090600+000001.00h.GRIB
Warning (cgribexScanTimestep1) : Param=65.2 level=0 already exist, skipped!
01:00:00
cdo showtime: Processed 34 variables over 1 timestep. ( 0.00s )_

Any suggestion to solve that is welcome.

A part these problems, I wish to thank all those who are contributing in writing CDOs.

Dario B. Giaiotti

ARPA FVG - CRMA
Centro Regionale di Modellistica Ambientale
Via Cairoli, 14
I-33057 Palmanova (UD)
ITALY
Room I/20/U
Tel +39 0432 191 8048
Certified e-mail - PEC arpa@certregione.fvg.it
e-mail dario.giaiotti@arpa.fvg.it
------------------------------------

@GRIBIT.F ---> Line 271
!
! CHUANG: TO OUTPUT OFF-HOUR FORECAST, I USED MIN INSTEAD OF HOUR AS FORECAST UNIT
! ALOS, SINCE ONLT TIME RANGE TYPE 10 USES 2 BYTES TO STORE TIME, MODIFICATION WAS
! MADE TO USE TYPE 10 AS TIME RANGE INDICATOE WHEN FORECST MINS ARE LARGER THAN 254,
! WHICH MEANS ALL THE ACCUMULATED AND TIME-AVERAGED QUANTITY ARE VERIFIED AT ONE TIME
! INSTEAD OF AT A TIME RANGE.
! J. HALLEY GOTWAY, MODIFY HOW THE TIME INFORMATION IS STORED IN ID,
! (FCST TIME UNIT, P1, P2, TIME RANGE INDICATOR).
! CHECK IF THE NUMBER OF FORECAST MINUTES IS ZERO FOR HOURS OR NON-ZERO FOR
! OFF-HOUR FORECASTS. FOR OFF-HOUR FORECASTS, CHECK IF THE TOTAL NUMBER
! OF MINUTES IS DIVISIBLE BY 30, 15, OR NEITHER, AND USE THE APPROPRIATE
! FCST TIME UNIT VALUE. FOR ANY FIELD OTHER THAN AN INSTANTANEOUS FIELD,
! ASSUME ID, ARE PASSED IN CORRECTLY.
!
! ---------------------------------------------------------------------------------
!
! By Dario B. Giaiotti on Aug 01, 2014
!
! ARPA FVG - CRMA
! Centro Regionale di Modellistica Ambientale
! Via Cairoli, 14
! I-33057 Palmanova (UD)
! ITALY
! Room I/20/U
! Tel +39 0432 191 8048
! Certified e-mail - PEC arpa@certregione.fvg.it
! e-mail dario.giaiotti@arpa.fvg.it
!
! Since CDO (https://code.zmaw.de/projects/cdo) apps
! do not accepts TIME RANGE INDICATOR 14 or 13 and all the post processing
! is based on CDo operators the OFF-HOUR FORECASTS concerning 30, 15, minutes
! are set to the default for minutes, i.e. 0

!
IFTHEN
! ID = 0
ID = 0 ! Added by Dario B. Giaiotti on Aug 01, 2014 to set TIME RANGE INDICATOR to minutes
DIV = 1 ! Added by Dario B. Giaiotti on Aug 01, 2014 to set TIME RANGE INDICATOR to minutes
! COMPUTE THE TOTAL FORECAST MINUTES.
TOTMIN=IFHR*60+IFMIN
!
! Block commented by Dario B. Giaiotti on Aug 01, 2014 to set TIME RANGE INDICATOR to minutes
!
! ================== BLOCK START ===============================================
!
! CHECK FOR 1/2 HOURLY INCREMENTS.
! IF (MOD == 0) THEN
! ID = 14
! DIV = 30
! CHECK FOR 1/4 HOURLY INCREMENTS.
! ELSEIF (MOD == 0) THEN
! ID = 13
! DIV = 15
! OTHERWISE, USE MINUTES.
! ELSE
! ID = 0
! DIV = 1
! ENDIF
!
! ================== BLOCK END ===============================================
!
! SET THE VALUES FOR P1 AND P2. USE TOTMIN FOR INSTANTANEOUS FIELDS.@

Replies (6)

RE: CDO operators and problems in reading GRIB1 sub hourly records generated by UPPV2.0 from WRF outputs - Added by Uwe Schulzweida almost 10 years ago

Could you please upload a small example file? That would help us to find a solution for this problem.

Cheers, Uwe

RE: CDO operators and problems in reading GRIB1 sub hourly records generated by UPPV2.0 from WRF outputs - Added by Giaiotti Dario almost 10 years ago

Dear Uwe,

many thanks for your availability in analyzing the problem.

Here attached you find a compressed archive (.tgz) storing GRIB1 records covering two hours of simulation with 10 minutes time resolution.
These records are already cleaned by the problem concerning "Time unit 14 unsupported", but they are still affected by the fact that CDOs do not recognize a single sub hourly record alone,

cdo sinfon CRMA_WRF-2-TESI_d03_2013090600+000000.10h.GRIB  
 Warning (cgribexScanTimestep2) : Param=229.2 level=10 not defined at timestep 1!
 cdo sinfon: Open failed on >CRMA_WRF-2-TESI_d03_2013090600+000000.10h.GRIB<
 Unsupported file structure

but it is recognized when it is concatenated with other GRIB1 records and the first is a sharp hour.

Here is the tail of cdo sinfon applied to all GRIB files concatenated, by means of cat command.

Horizontal grids :
     1 : lonlat       > size      : dim = 8100  nlon = 90  nlat = 90
                        lon       : first = 11.888  last = 14.113  inc = 0.025  degrees_east
                        lat       : first = 45.322  last = 46.835  inc = 0.017  degrees_north
   Vertical grids :
     1 : meansea            level : 0 
     2 : surface                  : 0 
     3 : pressure              Pa : 10000 20000 30000 40000 50000 60000 70000 80000 
                                    85000 90000 92500 95000 97500 100000 
     4 : height                 m : 2 
     5 : height                 m : 10 
     6 : generic      (ltype=214) : 0 
     7 : generic      (ltype=224) : 0 
     8 : generic      (ltype=234) : 0 
     9 : generic      (ltype=200) : 0 
   Time axis :  unlimited steps
     RefTime =  2013-09-06 00:00:00  Units = hours  Calendar = PROLEPTIC
  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss
  2013-09-06 00:00:00  2013-09-06 00:10:00  2013-09-06 00:20:00  2013-09-06 00:30:00
  2013-09-06 00:40:00  2013-09-06 00:50:00  2013-09-06 01:00:00  2013-09-06 01:10:00
  2013-09-06 01:20:00  2013-09-06 01:30:00  2013-09-06 01:40:00  2013-09-06 01:50:00
  2013-09-06 02:00:00  2013-09-06 02:10:00  2013-09-06 02:20:00  2013-09-06 02:30:00
  2013-09-06 02:40:00  2013-09-06 02:50:00
cdo sinfon: Processed 34 variables over 18 timesteps. ( 0.04s )

With best regards,

Dario Giaiotti

CRMA_WRF_d03_2013090600.tgz (16.4 MB) CRMA_WRF_d03_2013090600.tgz

RE: CDO operators and problems in reading GRIB1 sub hourly records generated by UPPV2.0 from WRF outputs - Added by Uwe Schulzweida almost 10 years ago

The CDO internal data model needs a GRIB file with a consistent record structure. That means each variable must occur only once and each time step needs to have the same variables. Your GRIB file contains the code 65 twice. One is accumulated and the other is instantaneous. This is not handled correctly by CDO. I have no workaround for this problem, sorry. I will try to implemented it in the next CDO release.
Could you also please upload an original file with the time unit 14? That will help me to implement the support for it.

RE: CDO operators and problems in reading GRIB1 sub hourly records generated by UPPV2.0 from WRF outputs - example with time unit 14 - Added by Giaiotti Dario almost 10 years ago

Dear Uwe,

I have attached an archive storing some GRIB1 files having the time unit code set to 14 for half hours, while the sharp hour and the minutes records have the standard WMO code. Here below I report some diagnostic made by means of wgrib (http://www.cpc.ncep.noaa.gov/products/wesley/wgrib.html) utility. I have highlighted in bold the octets we are talking about.

Concerning the files with GRIB1 records having the same value in the octet 9 (e.g. 65 twice in my example), i suppose the problem could be handled allowing CDOs to read more information on the section 1 of GRIB1.

Many thanks for your availability in considering to include these updates in future CDO releases.

With best regards,

Dario

$ wgrib -d 3 -V -PDS10 CRMA_WRF-2-TESI_d03_2013090600+000002.00h.GRIB
 rec 3:27526:date 2013090600 HGT kpds5=7 kpds6=100 kpds7=100 levels=(0,100) grid=255 100 mb 2hr fcst:
  HGT=Geopotential height [gpm]
  timerange 0 P1 2 P2 0 TimeU 1  nx 90 ny 90 GDS grid 3 num_in_ave 0 missing 0
  center 7 subcenter 0 process 125 Table 2 scan: WE:SN winds(grid) 
  Lambert Conf: Lat1 45.322000 Lon1 11.888000 Lov 11.300000
      Latin1 45.000000 Latin2 45.000000 LatSP 0.000000 LonSP 0.000000
      North Pole (90 x 90) Dx 2.000000 Dy 2.000000 scan 64 mode 136
  PDS10(1..28)=
       1:   0   0  28   2   7 125 255 128   7 100   0 100  13   9   6   0   0   1   2   0
      21:   0   0   0   0  21   0   0   4
  min/max data 16491.7 16514.5  num bits 16  BDS_Ref 1.64917e+08  DecScale 4 BinScale 2

$ wgrib -d 3 -V -PDS10 CRMA_WRF-2-TESI_d03_2013090600+000002.10h.GRIB
 rec 3:27526:date 2013090600 HGT kpds5=7 kpds6=100 kpds7=100 levels=(0,100) grid=255 100 mb 130min fcst:
  HGT=Geopotential height [gpm]
  timerange 10 P1 0 P2 130 TimeU 0  nx 90 ny 90 GDS grid 3 num_in_ave 0 missing 0
  center 7 subcenter 0 process 125 Table 2 scan: WE:SN winds(grid) 
  Lambert Conf: Lat1 45.322000 Lon1 11.888000 Lov 11.300000
      Latin1 45.000000 Latin2 45.000000 LatSP 0.000000 LonSP 0.000000
      North Pole (90 x 90) Dx 2.000000 Dy 2.000000 scan 64 mode 136
  PDS10(1..28)=
       1:   0   0  28   2   7 125 255 128   7 100   0 100  13   9   6   0   0   0   0 130
      21:  10   0   0   0  21   0   0   4
  min/max data 16489.9 16517.4  num bits 16  BDS_Ref 1.64899e+08  DecScale 4 BinScale 3

$ wgrib -d 3 -V -PDS10 CRMA_WRF-2-TESI_d03_2013090600+000002.30h.GRIB
 rec 3:27526:date 2013090600 HGT kpds5=7 kpds6=100 kpds7=100 levels=(0,100) grid=255 100 mb 150min fcst:
  HGT=Geopotential height [gpm]
  timerange 10 P1 0 P2 5 TimeU 14  nx 90 ny 90 GDS grid 3 num_in_ave 0 missing 0
  center 7 subcenter 0 process 125 Table 2 scan: WE:SN winds(grid) 
  Lambert Conf: Lat1 45.322000 Lon1 11.888000 Lov 11.300000
      Latin1 45.000000 Latin2 45.000000 LatSP 0.000000 LonSP 0.000000
      North Pole (90 x 90) Dx 2.000000 Dy 2.000000 scan 64 mode 136
  PDS10(1..28)=
       1:   0   0  28   2   7 125 255 128   7 100   0 100  13   9   6   0   0  14  0   5
      21:  10   0   0   0  21   0   0   4
  min/max data 16500.5 16520.9  num bits 16  BDS_Ref 1.65005e+08  DecScale 4 BinScale 2

CRMA_WRF-PDS-14_d03_2013090600.tgz (16.5 MB) CRMA_WRF-PDS-14_d03_2013090600.tgz

RE: CDO operators and problems in reading GRIB1 sub hourly records generated by UPPV2.0 from WRF outputs - Added by Uwe Schulzweida over 9 years ago

Thanks for the GRIB files. The next CDO version will read them correctly!

Cheers, Uwe

RE: CDO operators and problems in reading GRIB1 sub hourly records generated by UPPV2.0 from WRF outputs - Added by Giaiotti Dario over 9 years ago

Dear Uwe,

good work and thanks for considering such improvement in the CDOs.
With best regards,

Dario

(1-6/6)

Project

General

Profile

CDO