merging multiple .nc4 files
Added by Kushank Bajaj over 7 years ago
Hi everyone!
I'm new to netcdf data and have been using cdo and nco since last two weeks. I'm trying to working with GLDAS data and facing a couple of issues.
While using mergetime operation on multiple .nc4 files, I get an error saying 'setBaseTime : Unsupported TIMEUNIT: days since 2001-03-01 00:00:00!'. I do get an output file which goes in an infinite loop and doesn't have a timeseries data.
I've tried the following operations:
cdo mergetime *.nc4 abc.nc4
cdo -f nc4 mergetime *.nc4 ofile.nc4
cdo -a mergetime *.nc4 ofile.nc4
Thanks in advance.
Replies (10)
RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer over 7 years ago
Hi Kushank,
did you try
rm -rf ofile.nc4 cdo -r -f nc4 -mergetime *.nc4 ofile.nc4
Can you upload some test files that we are able to see what's going on.
-Karin
RE: merging multiple .nc4 files - Added by Kushank Bajaj over 7 years ago
Hi Karin,
Thanks for your response. I tried what you suggested. Getting the same error.
I've attached one of the monthly-input file that I'm trying to merge. All the other input fills are similar.
I tried to merge two files which turned out to be 270mb in size and when i checked for number of months using -cdo nmon ifile.nc4; it shows only 1 month.
I guess the time is not specified and I'm a little clueless how to define it.
Also, could you please suggest if it's possible to separate multiple variables out of all these input files without merging them?
Thanks!
RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer over 7 years ago
Is the date of you files always 0000-00-00 00:00:00 ? This could be an explaination why you are getting an error using the mergetime operator.
cdo info Tair_f_inst.A200001.021.nc
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 0000-00-00 00:00:00 0 864000 620117 : 227.61 272.13 305.93 : -1
cdo info: Processed 864000 values from 1 variable over 1 timestep ( 0.02s )
[~/CDO/Support/time] > cdo info GLDAS_NOAH025_M.A200001.021.nc4
setBaseTime : Unsupported TIMEUNIT: days since 2001-03-01 00:00:00!
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 91.468 333.83 : -1
2 : 0000-00-00 00:00:00 0 864000 620117 : -138.33 -50.345 21.526 : -2
3 : 0000-00-00 00:00:00 0 864000 620117 : -8.1117 25.184 198.23 : -3
4 : 0000-00-00 00:00:00 0 864000 620117 : -90.358 19.755 224.48 : -4
5 : 0000-00-00 00:00:00 0 864000 620117 : -38.133 -3.6613 10.942 : -5
6 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 6.9819e-06 0.00019088 : -6
7 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 1.6608e-05 0.00020218 : -7
8 : 0000-00-00 00:00:00 0 864000 620117 : -2.8735e-06 1.0012e-05 7.9258e-05 : -8
9 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 0.0088173 1.0727 : -9
10 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 0.052422 1.9312 : -10
11 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 0.0088973 1.2122 : -11
12 : 0000-00-00 00:00:00 0 864000 620117 : 222.80 271.75 308.96 : -12
13 : 0000-00-00 00:00:00 0 864000 620117 : 5.0000 38.533 80.806 : -13
14 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 806.83 1.0180e+05 : -14
15 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 2.1524 254.50 : -15
16 : 0000-00-00 00:00:00 0 864000 620117 : 2.0000 25.096 47.572 : -16
17 : 0000-00-00 00:00:00 0 864000 620117 : 6.0574 75.123 140.40 : -17
18 : 0000-00-00 00:00:00 0 864000 620117 : 12.000 139.03 279.00 : -18
19 : 0000-00-00 00:00:00 0 864000 620117 : 20.000 265.26 476.00 : -19
20 : 0000-00-00 00:00:00 0 864000 620117 : 238.09 275.81 308.90 : -20
21 : 0000-00-00 00:00:00 0 864000 620117 : 238.52 277.76 307.84 : -21
22 : 0000-00-00 00:00:00 0 864000 620117 : 239.38 278.90 307.08 : -22
23 : 0000-00-00 00:00:00 0 864000 620117 : 240.31 280.58 306.04 : -23
24 : 0000-00-00 00:00:00 0 864000 620117 : -8.1117 96.691 631.26 : -24
25 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 6.9961 85.443 : -25
26 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 9.5479 124.24 : -26
27 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 7.6910 114.27 : -27
28 : 0000-00-00 00:00:00 0 864000 620117 : 2.0000 277.23 896.50 : -28
29 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 0.15641 0.50000 : -29
30 : 0000-00-00 00:00:00 0 864000 620117 : 0.61588 3.6337 12.538 : -30
31 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 2.3590e-05 0.00020218 : -31
32 : 0000-00-00 00:00:00 0 864000 620117 : 227.61 272.13 305.93 : -32
33 : 0000-00-00 00:00:00 0 864000 620117 : 9.6237e-05 0.0052794 0.020951 : -33
34 : 0000-00-00 00:00:00 0 864000 620117 : 45068. 93849. 1.0524e+05 : -34
35 : 0000-00-00 00:00:00 0 864000 620117 : 0.0000 116.79 389.65 : -35
36 : 0000-00-00 00:00:00 0 864000 620117 : 121.25 268.60 456.12 : -36
cdo info: Processed 31104000 values from 36 variables over 1 timestep ( 0.64s )To extract a single variable from a file, e.g. Tair_f_inst
cdo -r -selvar,Tair_f_inst GLDAS_NOAH025_M.A200001.021.nc4 Tair_f_inst.A200001.021.ncor multiple variables
cdo -r -selvar,Tair_f_inst,Psurf_f_inst,SnowDepth_inst GLDAS_NOAH025_M.A200001.021.nc4 GLDAS_NOAH025_M-Tair_f_inst-Psurf_f_inst-SnowDepth.A200001.021.nc4
-Karin
RE: merging multiple .nc4 files - Added by Kushank Bajaj over 7 years ago
Hi Karin,
Thanks for helping. Yes, the time is always 0000-00-00.
Is there any solution for that? There are about 200 similar files.
To extract multiple variables I used the same operation and it worked fine. I was wondering if I could extract multiple variables from multiple files at the same time. I used
$ cdo -f nc selvar,SoilMoi0_10cm_inst,SoilMoi100_200cm_inst,SoilMoi10_40cm_inst,SoilMoi40_100cm_inst *.nc4 test1.nc
which shows the following error : cdo selname (Abort): Too many streams! Operator needs 1 input and 1 output streams.
Thanks.
RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer over 7 years ago
To select variables from multiple files use the select operator
cdo select,name=var1,var2,...,varN '*.nc4' outputfile.nc4
If you know the time value for each file you can use the settaxis operator, e.g. for monthly data 1987-01-16 12:00:00
cdo -setreftime,1980-01-01,12:00:00 -settaxis,1987-01-16,12:00:00,1mon infile outfile
The best way would be to create the data with the correct time dimension from the very first. 
-Karin
RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer over 7 years ago
Ok, settaxis doesn't work, too. The time units attribute seems to be corrupted (blank in front of days). To correct the time units you can use NCO's ncatted
ncatted -h -O -a units,time,o,c,"days since 2001-03-01 00:00:00" GLDAS_NOAH025_M.A200001.021.nc4 tmp.A200001.021.nc
The CDO's are working good with the tmp.A200001.021.nc file.
-Karin
RE: merging multiple .nc4 files - Added by Kushank Bajaj over 7 years ago
Hi Karin,
Thanks a lot!!! ncatted works fine .
But I've got about 210 files. Do you think it's possible to do this for all the files together?
-Kushank
RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer over 7 years ago
Hi Kushank,
I recommend to use a shell script .e.g. Korn-Shell script
#!/bin/ksh
infiles=$(ls *.nc4)
for file in ${infiles}
do
fbase=${file%.*}
ncatted -h -O -a units,time,o,c,"days since 2001-03-01 00:00:00" ${file} ${fbase}_correct.nc4
done
exit
-Karin
RE: merging multiple .nc4 files - Added by Kushank Bajaj over 7 years ago
Hi Karin,
Thank you soooooo much!! you made my day!
RE: merging multiple .nc4 files - Added by Kushank Bajaj over 7 years ago
Karin,
I have another query regarding masking region from .nc files.
Would you happen to know how to mask a region from netcdf file using a shape/netcdf file?
Thanks!
-Kushank