Project

General

Profile

Forums » Support »

error extracting data from UCAR NCEP grib file: error msg: "inconsistent timestep"

Added by Ian Scott-Fleming almost 12 years ago

Have downloaded NCEP DS090 data from UCAR website ( http rda.ucar.edu/cgi-bin/datasets/getWebList ), and get the following error msg when trying to extract the variables and/or pressure levels I need:

cdo -f nc selname,var7,var11,var52 A15094-197701.pgb.f00 t01.nc

Error (cgribexScanTimestep) : Inconsistent timestep 125 (GRIB record 2/140)!

or:

cdo -O -f nc sellevel,85000,70000,60000,50000 A15094-197701.pgb.f00 t01.nc

Error (cgribexScanTimestep) : Inconsistent timestep 125 (GRIB record 2/140)!

Files are grib files, and seem to contain many variables, levels, and possibly multiple time sequences?

cdo showlevel and cdo showname work with no error msgs, but others, such as cdo sinfo also abort with same error msg.

Any suggestions would be most appreciated.

Data files are too large to upload (250 MB) , but have put one on web at http icsf-jmac.ttu.edu/ucar_data/ for download if helpful.

Data files were downloaded from UCAR via wget script generated by their web interface. Similar files downloaded via curl script do not generate this error, but I need to use the ones downloaded via wget, as the curl ones seem to have some inconsistencies and am trying to determine where those problems occur. sigh. :(

cdo sinfon reports the following for the file (with same error msg at bottom):

cdo sinfon A15094-197701.pgb.f00 t01.nc
File format: GRIB
-1 : Institut Source Name Time Typ Grid Size Num Levels Num
1 : NCEP T62L28MRFvar7 var P12 10512 1 17 1
2 : NCEP T62L28MRFvar33 var P11 10512 1 17 1
3 : NCEP T62L28MRFvar34 var P11 10512 1 17 1
4 : NCEP T62L28MRFvar11 var P10 10512 1 17 1
5 : NCEP T62L28MRFvar39 var P11 10512 1 12 2
6 : NCEP T62L28MRFvar52 var P7 10512 1 8 3
7 : NCEP T62L28MRFvar41 var P13 10512 1 17 1
8 : NCEP T62L28MRFvar33 var P9 10512 1 1 4
9 : NCEP T62L28MRFvar34 var P9 10512 1 1 4
10 : NCEP T62L28MRFvar11 var P10 10512 1 1 4
11 : NCEP T62L28MRFvar52 var P7 10512 1 1 4
12 : NCEP T62L28MRFvar33 var P10 10512 1 3 5
13 : NCEP T62L28MRFvar34 var P9 10512 1 3 5
14 : NCEP T62L28MRFvar1 var P13 10512 1 1 6
15 : NCEP T62L28MRFvar54 var P10 10512 1 1 7
16 : NCEP T62L28MRFvar52 var P7 10512 1 1 7
17 : NCEP T62L28MRFvar11 var P9 10512 1 1 8
18 : NCEP T62L28MRFvar1 var P12 10512 1 1 8
19 : NCEP T62L28MRFvar33 var P11 10512 1 1 8
20 : NCEP T62L28MRFvar34 var P10 10512 1 1 8
21 : NCEP T62L28MRFvar136 var P9 10512 1 1 8
22 : NCEP T62L28MRFvar131 var P9 10512 1 1 6
23 : NCEP T62L28MRFvar132 var P9 10512 1 1 6
24 : NCEP T62L28MRFvar11 var P10 10512 1 1 9
25 : NCEP T62L28MRFvar1 var P13 10512 1 1 9
26 : NCEP T62L28MRFvar33 var P11 10512 1 1 9
27 : NCEP T62L28MRFvar34 var P11 10512 1 1 9
28 : NCEP T62L28MRFvar7 var P13 10512 1 1 6
29 : NCEP T62L28MRFvar2 var P10 10512 1 1 10
30 : NCEP T62L28MRFvar52 var P7 10512 1 3 11
31 : NCEP T62L28MRFvar13 var P10 10512 1 1 12
32 : NCEP T62L28MRFvar11 var P10 10512 1 1 12
33 : NCEP T62L28MRFvar39 var P12 10512 1 1 12
34 : NCEP T62L28MRFvar52 var P7 10512 1 1 12
35 : NCEP T62L28MRFvar33 var P9 10512 1 1 12
36 : NCEP T62L28MRFvar34 var P9 10512 1 1 12
Horizontal grids :
1 : lonlat > size : dim = 10512 nlon = 144 nlat = 73
lon : first = 0 last = 357.5 inc = 2.5 degrees_east circular
lat : first = 90 last = -90 inc = -2.5 degrees_north
Vertical grids :
1 : pressure Pa : 100000 92500 85000 70000 60000 50000 40000 30000
25000 20000 15000 10000 7000 5000 3000 2000 1000
2 : pressure Pa : 100000 92500 85000 70000 60000 50000 40000 30000
25000 20000 15000 10000
3 : pressure Pa : 100000 92500 85000 70000 60000 50000 40000 30000
4 : generic (ltype=116) : 30
5 : altitude m : 1829 2743 3658
6 : surface : 0
7 : generic (ltype=200) : 0
8 : generic (ltype= 7) : 0
9 : generic (ltype= 6) : 0
10 : meansea level : 0
11 : sigma level : 72 58 83
bounds : 44-100 44-72 72-94
12 : sigma level : 9950
Time axis : unlimited steps
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
1977-01-01 00:00:00 1977-01-01 06:00:00 1977-01-01 12:00:00 1977-01-01 18:00:00
1977-01-02 00:00:00 1977-01-02 06:00:00 1977-01-02 12:00:00 1977-01-02 18:00:00
1977-01-03 00:00:00 1977-01-03 06:00:00 1977-01-03 12:00:00 1977-01-03 18:00:00
1977-01-04 00:00:00 1977-01-04 06:00:00 1977-01-04 12:00:00 1977-01-04 18:00:00
1977-01-05 00:00:00 1977-01-05 06:00:00 1977-01-05 12:00:00 1977-01-05 18:00:00
1977-01-06 00:00:00 1977-01-06 06:00:00 1977-01-06 12:00:00 1977-01-06 18:00:00
1977-01-07 00:00:00 1977-01-07 06:00:00 1977-01-07 12:00:00 1977-01-07 18:00:00
1977-01-08 00:00:00 1977-01-08 06:00:00 1977-01-08 12:00:00 1977-01-08 18:00:00
1977-01-09 00:00:00 1977-01-09 06:00:00 1977-01-09 12:00:00 1977-01-09 18:00:00
1977-01-10 00:00:00 1977-01-10 06:00:00 1977-01-10 12:00:00 1977-01-10 18:00:00
1977-01-11 00:00:00 1977-01-11 06:00:00 1977-01-11 12:00:00 1977-01-11 18:00:00
1977-01-12 00:00:00 1977-01-12 06:00:00 1977-01-12 12:00:00 1977-01-12 18:00:00
1977-01-13 00:00:00 1977-01-13 06:00:00 1977-01-13 12:00:00 1977-01-13 18:00:00
1977-01-14 00:00:00 1977-01-14 06:00:00 1977-01-14 12:00:00 1977-01-14 18:00:00
1977-01-15 00:00:00 1977-01-15 06:00:00 1977-01-15 12:00:00 1977-01-15 18:00:00
1977-01-16 00:00:00 1977-01-16 06:00:00 1977-01-16 12:00:00 1977-01-16 18:00:00
1977-01-17 00:00:00 1977-01-17 06:00:00 1977-01-17 12:00:00 1977-01-17 18:00:00
1977-01-18 00:00:00 1977-01-18 06:00:00 1977-01-18 12:00:00 1977-01-18 18:00:00
1977-01-19 00:00:00 1977-01-19 06:00:00 1977-01-19 12:00:00 1977-01-19 18:00:00
1977-01-20 00:00:00 1977-01-20 06:00:00 1977-01-20 12:00:00 1977-01-20 18:00:00
1977-01-21 00:00:00 1977-01-21 06:00:00 1977-01-21 12:00:00 1977-01-21 18:00:00
1977-01-22 00:00:00 1977-01-22 06:00:00 1977-01-22 12:00:00 1977-01-22 18:00:00
1977-01-23 00:00:00 1977-01-23 06:00:00 1977-01-23 12:00:00 1977-01-23 18:00:00
1977-01-24 00:00:00 1977-01-24 06:00:00 1977-01-24 12:00:00 1977-01-24 18:00:00
1977-01-25 00:00:00 1977-01-25 06:00:00 1977-01-25 12:00:00 1977-01-25 18:00:00
1977-01-26 00:00:00 1977-01-26 06:00:00 1977-01-26 12:00:00 1977-01-26 18:00:00
1977-01-27 00:00:00 1977-01-27 06:00:00 1977-01-27 12:00:00 1977-01-27 18:00:00
1977-01-28 00:00:00 1977-01-28 06:00:00 1977-01-28 12:00:00 1977-01-28 18:00:00
1977-01-29 00:00:00 1977-01-29 06:00:00 1977-01-29 12:00:00 1977-01-29 18:00:00
1977-01-30 00:00:00 1977-01-30 06:00:00 1977-01-30 12:00:00 1977-01-30 18:00:00
1977-01-31 00:00:00 1977-01-31 06:00:00 1977-01-31 12:00:00 1977-01-31 18:00:00
Error (cgribexScanTimestep) : Inconsistent timestep 125 (GRIB record 2/140)!


Replies (4)

RE: error extracting data from UCAR NCEP grib file: error msg: "inconsistent timestep" - Added by Jaison-Thomas Ambadan almost 12 years ago

Hi,

I didn't find any problem(s); both in "cdo sinfo" as well as:

============================

cdo -f nc selname,var7,var11,var52 A00294-198601.pgb.f00 t01.nc
cdo selname: Processed 69084864 values from 36 variables over 124 timesteps. ( 2.08s )

============================

I'm using the latest cdo v1.5.5 (with CGRIBEX as well as GRIB_API). I think the problem is the with the GRIB library that you've compiled-in with your cdo; so please post your "cdo -V"

Cheers,
J.

RE: error extracting data from UCAR NCEP grib file: error msg: "inconsistent timestep" - Added by Ian Scott-Fleming almost 12 years ago

Looks like there may be a problem with the 1977 data file I posted...my explanation is below the "_____"

Looks like I'm at 1.5.4:

~ $ cdo -V
Climate Data Operators version 1.5.4 (http://code.zmaw.de/projects/cdo)
Compiler: /usr/bin/llvm-gcc-4.2 -std=gnu99 -pipe -O2 -arch x86_64 -D_THREAD_SAFE -pthread
version: i686-apple-darwin10-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.9)
with: PTHREADS NC4 OPeNDAP SZ Z PROJ.4
Compiled: by root on icsf_jmac (x86_64-apple-darwin10.8.0) May 15 2012 11:28:23
CDI library version : 1.5.4 of May 15 2012 11:28:21
CGRIBEX library version : 1.5.1 of Aug 29 2011 20:30:27
netCDF library version : 4.2 of May 19 2012 21:51:04 $
HDF5 library version : 1.8.9
SERVICE library version : 1.3.0 of May 15 2012 11:28:18
EXTRA library version : 1.3.0 of May 15 2012 11:28:18
IEG library version : 1.3.0 of May 15 2012 11:28:18
FILE library version : 1.7.1 of May 15 2012 11:28:18

Also looks like I missed the last sentence when I cut/pasted my original post...I left out the sentences: "Have posted two files, from 1977 & 1986. I don't get the error on the 1986 data, but I do on the 1977 data".

Here's my output from the two files:

s/ucar_data $ ls
A00294-198601.pgb.f00 A15094-197701.pgb.f00
icsf_jmac:iscottfl:/Library/WebServer/Documents/ucar_data $ cdo -f nc selname,var7,var11,var52 A00294-198601.pgb.f00 t01.nc
cdo selname: Processed 69084864 values from 36 variables over 124 timesteps. ( 1.26s )
icsf_jmac:iscottfl:/Library/WebServer/Documents/ucar_data $ cdo -f nc selname,var7,var11,var52 A15094-197701.pgb.f00 t01.nc

Error (cgribexScanTimestep) : Inconsistent timestep 125 (GRIB record 2/140)!

Not being too familiar with grib files, I didn't understand the error message, and assumed cdo was saying there was an error in record 2 in the file. However, on closer inspection, it seems to be saying an error with record 2 of the 125th timestep. Since the data file contains data for 31 days, 4 hours/day, there should only be 124 timesteps (140 records/timestep, so 17360 recs total). A wgrib dump of the file shows, for the Jan 1977 data:

17358:232530610:D=1977013118:RH:sigma=0.9950:kpds=52,107,9950:anl:winds are N/S:"Relative humidity [%]
17359:232539892:D=1977013118:UGRD:sigma=0.9950:kpds=33,107,9950:anl:winds are N/S:"u wind [m/s]
17360:232551802:D=1977013118:VGRD:sigma=0.9950:kpds=34,107,9950:anl:winds are N/S:"v wind [m/s]
17361:232566008:D=1977013118:HGT:700 mb:kpds=7,100,700:anl:winds are N/S:"Geopotential height [gpm]
17362:232579232:D=1977013118:HGT:600 mb:kpds=7,100,600:anl:winds are N/S:"Geopotential height [gpm]
icsf_jmac:iscottfl:/Library/WebServer/Documents/ucar_data $

while for the Jan 1986 data:

17358:233838086:D=1986013118:RH:sigma=0.9950:kpds=52,107,9950:anl:winds are N/S:"Relative humidity [%]
17359:233847368:D=1986013118:UGRD:sigma=0.9950:kpds=33,107,9950:anl:winds are N/S:"u wind [m/s]
17360:233859278:D=1986013118:VGRD:sigma=0.9950:kpds=34,107,9950:anl:winds are N/S:"v wind [m/s]
icsf_jmac:iscottfl:/Library/WebServer/Documents/ucar_data $

I.e., two spurious extra lines at the end of the 1977 data file. I haven't figured out yet how to determine whether the two extra records match the earlier ones in the file or not.

What's puzzling me is how can I be the first to have encountered this error with the UCAR data? I have downloaded data from UCAR for 1948 to present, and many of the files report this error. I'm guessing that perhaps other users of this data are using other utilities than cdo which either ignore the extra lines or don't produce an error message. (My coworkers all use nco utilities, not cdo.)

cdo produces netcdf output from both files, but the 1977 output file has a single timestep, and all data is set to 0000 hrs on 1/1/1977, while the 1986 file is correctly tagged with 124 timesteps. I have added the output files to the website.

There's clearly a bug in the data, and I will contact UCAR regarding this. The question is whether cdo's behaviour here should be considered a bug.

It would be easy to ignore cdo's error message and use the output file in a large analysis. I have nearly 1000 files I'm working with, so would normally write scripts to process them all, and miss the error messages buried in all the other output.

RE: error extracting data from UCAR NCEP grib file: error msg: "inconsistent timestep" - Added by Jaison-Thomas Ambadan almost 12 years ago

Hi,

cdo produces netcdf output from both files, but the 1977 output file has a single timestep

The "single timestep" problem can be solved, if do the same procedure in two steps, i,e,

1. cdo -copy A15094-197701.pgb.f00 test.grb
2. cdo -f nc -copy test.grb test.nc

The first step will decode and encode all the grid records one by one (all the 124 out of 125 steps, although it shows the error). It doesn't work in a single command sequence (in this case CDO will encode the NetCDF file with the first time step, because of the error) - may be because of the way CDO handles the NetCDF (I'm not an expert either). Anyway, hope this helps!

Cheers,
J.

RE: error extracting data from UCAR NCEP grib file: error msg: "inconsistent timestep" - Added by Uwe Schulzweida almost 12 years ago

I will change the Error/Abort to a Warning message in the next CDO release. Then the first 124 timesteps are processed correctly and the last 2 records are ignored. The changes will be available tomorrow in the current release: https://code.zmaw.de/projects/cdo/files

Here is an example on how to delete the last 2 records with wgrib:

wgrib A15094-197701.pgb.f00 > inventory_file
edit the inventory_file and delete the last 2 lines
wgrib -i -grib -o A15094-197701.pgb.f00.new A15094-197701.pgb.f00 < inventory_file

    (1-4/4)