Problem merging large GRIB files?
Added by Jan Elbers about 13 years ago
Hello,
I try to merge 3 GRIB files (1.7 Gb each) with the command
cdo mergetime T_RH_NH_1999_1 T_RH_NH_1999_2 T_RH_NH_1999_3 test
and get the following message: "Processed 6 variables over 1464 timesteps"
The number of timesteps is correct (3*488) however the 3 files contain 2 variables, each for 7 pressure levels
Here's the ctl file for the first file:
index ^T_RH_NH_1999_1.idx
undef 9.999E+20
title T_RH_NH_1999_1
- produced by grib2ctl v0.9.12.5p45
dtype grib 255
options yrev
ydef 181 linear 0.000000 0.5
xdef 720 linear -180.000000 0.500000
tdef 488 linear 03Z01apr1999 3hr - z has 7 levels, for prs
zdef 7 levels
1000 950 900 850 800 750 700
vars 2
Rprs 7 157,100,0 * (profile) Relative humidity %
Tprs 7 130,100,0 * (profile) Temperature K
ENDVARS
The merged file seems correct in terms of size but there are problems with the file structure.
What am i doing wrong?
By the way, if the outputfile already exists and i say yes to the overwrite question CDO aborts with the message "Outputfile test already exist!"
Replies (3)
RE: Problem merging large GRIB files? - Added by Uwe Schulzweida about 13 years ago
Hi Jan,
The processed message is not very useful when processing more than one input file. Counted are simple all read variables and all read time steps. In this case 3*2 variables and 3*488 timesteps.
I can't reproduce a problem with mergetime. What kind of problem do you have with the file structure?
I also can't reproduce the result of the overwrite question. Which CDO version are you using?
Best regards,
Uwe
RE: Problem merging large GRIB files? - Added by Jan Elbers about 13 years ago
Hi Uwe,
Thanks for your reply. I use version 1.4.7.
I tried again but now on a UNIX machine and there it works fine. Could there be memory issues with CDO under Windows 32-bit?
I also tried to merge 3 Netcdf files into one with the command (also in Windows):
cdo mergetime T_RH_SH_1999_1.nc T_RH_SH_1999_2.nc T_RH_SH_1999_3.nc T_RH_SH_1999.nc
I get an error message:
Assertion failed: pxp->bf_offset <= offset && offset < pxp->bf_offset + (off_t)
pxp->bf_extend, file posixio.c, line 325
This happens when the output file size reaches 2 Gb, again a memory issue?
Best Regards,
Jan
RE: Problem merging large GRIB files? - Added by Jan Elbers about 13 years ago
Just tried to merge the 3 Netcdf files on a Unix machine. The merge process stops at about 3 Gb. The files i try to merge are 1.7 Gb each. Used version 1.4.4. Is there a file size limit on UNIX too?
Best Regards,
Jan