merging multiple .nc4 files

Added by Kushank Bajaj almost 7 years ago

Hi everyone!
I'm new to netcdf data and have been using cdo and nco since last two weeks. I'm trying to working with GLDAS data and facing a couple of issues.

While using mergetime operation on multiple .nc4 files, I get an error saying 'setBaseTime : Unsupported TIMEUNIT: days since 2001-03-01 00:00:00!'. I do get an output file which goes in an infinite loop and doesn't have a timeseries data.

I've tried the following operations:
cdo mergetime *.nc4 abc.nc4
cdo -f nc4 mergetime *.nc4 ofile.nc4
cdo -a mergetime *.nc4 ofile.nc4

Thanks in advance.

Replies (10)

RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer almost 7 years ago

Hi Kushank,

did you try

rm -rf ofile.nc4
cdo -r -f nc4 -mergetime *.nc4 ofile.nc4

Can you upload some test files that we are able to see what's going on.

-Karin

RE: merging multiple .nc4 files - Added by Kushank Bajaj almost 7 years ago

Hi Karin,

Thanks for your response. I tried what you suggested. Getting the same error.
I've attached one of the monthly-input file that I'm trying to merge. All the other input fills are similar.

I tried to merge two files which turned out to be 270mb in size and when i checked for number of months using -cdo nmon ifile.nc4; it shows only 1 month.

I guess the time is not specified and I'm a little clueless how to define it.

Also, could you please suggest if it's possible to separate multiple variables out of all these input files without merging them?

Thanks!

GLDAS_NOAH025_M.A200001.021.nc4 (24.1 MB) GLDAS_NOAH025_M.A200001.021.nc4

RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer almost 7 years ago

Is the date of you cdo info Tair_f_inst.A200001.021.nc

-1 : 1 : 0000-00-00 00:00:00 cdo info: Processed [~/CDO/Support/time] setBaseTime -1 : Date 1 : 0000-00-00 00:00:00 2 : 0000-00-00 00:00:00 3 : 0000-00-00 00:00:00 4 : 0000-00-00 00:00:00 5 : 0000-00-00 00:00:00 6 : 0000-00-00 00:00:00 7 : 0000-00-00 00:00:00 8 : 0000-00-00 00:00:00 9 : 0000-00-00 00:00:00 10 : 0000-00-00 00:00:00 11 : 0000-00-00 00:00:00 12 : 0000-00-00 00:00:00 13 : 0000-00-00 00:00:00 14 : 0000-00-00 00:00:00 15 : 0000-00-00 00:00:00 16 : 0000-00-00 00:00:00 17 : 0000-00-00 00:00:00 18 : 0000-00-00 00:00:00 19 : 0000-00-00 00:00:00 20 : 0000-00-00 00:00:00 21 : 0000-00-00 00:00:00 22 : 0000-00-00 00:00:00 23 : 0000-00-00 00:00:00 24 : 0000-00-00 00:00:00 25 : 0000-00-00 00:00:00 26 : 0000-00-00 00:00:00 27 : 0000-00-00 00:00:00 28 : 0000-00-00 00:00:00 29 : 0000-00-00 00:00:00 30 : 0000-00-00 00:00:00 31 : 0000-00-00 00:00:00 32 : 0000-00-00 00:00:00 33 : 0000-00-00 00:00:00 34 : 0000-00-00 00:00:00 35 : 0000-00-00 00:00:00 36 : 0000-00-00 00:00:00 cdo info: Processed files always 0000-00-00 00:00:00 ? This could be an explaination why you are getting an error using the mergetime operator.

 Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter ID 0   864000  620117 :      227.61      272.13      305.93 : -1 864000 values from 1 variable over 1 timestep ( 0.02s ) > cdo info GLDAS_NOAH025_M.A200001.021.nc4 : Unsupported TIMEUNIT:  days since 2001-03-01 00:00:00! Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter ID 0   864000  620117 :      0.0000      91.468      333.83 : -1 0   864000  620117 :     -138.33     -50.345      21.526 : -2 0   864000  620117 :     -8.1117      25.184      198.23 : -3 0   864000  620117 :     -90.358      19.755      224.48 : -4 0   864000  620117 :     -38.133     -3.6613      10.942 : -5 0   864000  620117 :      0.0000  6.9819e-06  0.00019088 : -6 0   864000  620117 :      0.0000  1.6608e-05  0.00020218 : -7 0   864000  620117 : -2.8735e-06  1.0012e-05  7.9258e-05 : -8 0   864000  620117 :      0.0000   0.0088173      1.0727 : -9 0   864000  620117 :      0.0000    0.052422      1.9312 : -10 0   864000  620117 :      0.0000   0.0088973      1.2122 : -11 0   864000  620117 :      222.80      271.75      308.96 : -12 0   864000  620117 :      5.0000      38.533      80.806 : -13 0   864000  620117 :      0.0000      806.83  1.0180e+05 : -14 0   864000  620117 :      0.0000      2.1524      254.50 : -15 0   864000  620117 :      2.0000      25.096      47.572 : -16 0   864000  620117 :      6.0574      75.123      140.40 : -17 0   864000  620117 :      12.000      139.03      279.00 : -18 0   864000  620117 :      20.000      265.26      476.00 : -19 0   864000  620117 :      238.09      275.81      308.90 : -20 0   864000  620117 :      238.52      277.76      307.84 : -21 0   864000  620117 :      239.38      278.90      307.08 : -22 0   864000  620117 :      240.31      280.58      306.04 : -23 0   864000  620117 :     -8.1117      96.691      631.26 : -24 0   864000  620117 :      0.0000      6.9961      85.443 : -25 0   864000  620117 :      0.0000      9.5479      124.24 : -26 0   864000  620117 :      0.0000      7.6910      114.27 : -27 0   864000  620117 :      2.0000      277.23      896.50 : -28 0   864000  620117 :      0.0000     0.15641     0.50000 : -29 0   864000  620117 :     0.61588      3.6337      12.538 : -30 0   864000  620117 :      0.0000  2.3590e-05  0.00020218 : -31 0   864000  620117 :      227.61      272.13      305.93 : -32 0   864000  620117 :  9.6237e-05   0.0052794    0.020951 : -33 0   864000  620117 :      45068.      93849.  1.0524e+05 : -34 0   864000  620117 :      0.0000      116.79      389.65 : -35 0   864000  620117 :      121.25      268.60      456.12 : -36 31104000 values from 36 variables over 1 timestep ( 0.64s )

To extract a single variable from a file, e.g. Tair_f_inst

cdo -r -selvar,Tair_f_inst  GLDAS_NOAH025_M.A200001.021.nc4  Tair_f_inst.A200001.021.nc

or multiple variables

cdo -r -selvar,Tair_f_inst,Psurf_f_inst,SnowDepth_inst  GLDAS_NOAH025_M.A200001.021.nc4  GLDAS_NOAH025_M-Tair_f_inst-Psurf_f_inst-SnowDepth.A200001.021.nc4

-Karin

RE: merging multiple .nc4 files - Added by Kushank Bajaj almost 7 years ago

Hi Karin,

Thanks for helping. Yes, the time is always 0000-00-00.

Is there any solution for that? There are about 200 similar files.

To extract multiple variables I used the same operation and it worked fine. I was wondering if I could extract multiple variables from multiple files at the same time. I used
$ cdo -f nc selvar,SoilMoi0_10cm_inst,SoilMoi100_200cm_inst,SoilMoi10_40cm_inst,SoilMoi40_100cm_inst *.nc4 test1.nc

which shows the following error : cdo selname (Abort): Too many streams! Operator needs 1 input and 1 output streams.

Thanks.

RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer almost 7 years ago

To select variables from multiple files use the select operator

cdo select,name=var1,var2,...,varN  '*.nc4'  outputfile.nc4

If you know the time value for each file you can use the settaxis operator, e.g. for monthly data 1987-01-16 12:00:00

cdo -setreftime,1980-01-01,12:00:00 -settaxis,1987-01-16,12:00:00,1mon infile outfile

The best way would be to create the data with the correct time dimension from the very first.

-Karin

RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer almost 7 years ago

Ok, settaxis doesn't work, too. The time units attribute seems to be corrupted (blank in front of days). To correct the time units you can use NCO's ncatted

ncatted -h -O -a units,time,o,c,"days since 2001-03-01 00:00:00" GLDAS_NOAH025_M.A200001.021.nc4 tmp.A200001.021.nc

The CDO's are working good with the tmp.A200001.021.nc file.

-Karin

RE: merging multiple .nc4 files - Added by Kushank Bajaj almost 7 years ago

Hi Karin,

Thanks a lot!!! ncatted works fine .
But I've got about 210 files. Do you think it's possible to do this for all the files together?

-Kushank

RE: merging multiple .nc4 files - Added by Karin Meier-Fleischer almost 7 years ago

Hi Kushank,

I recommend to use a shell script .e.g. Korn-Shell script

#!/bin/ksh

  infiles=$(ls *.nc4)

  for file in ${infiles}
  do
     fbase=${file%.*}
     ncatted -h -O -a units,time,o,c,"days since 2001-03-01 00:00:00" ${file} ${fbase}_correct.nc4
  done

exit

-Karin

RE: merging multiple .nc4 files - Added by Kushank Bajaj almost 7 years ago

Hi Karin,

Thank you soooooo much!! you made my day!

RE: merging multiple .nc4 files - Added by Kushank Bajaj almost 7 years ago

Karin,

I have another query regarding masking region from .nc files.

Would you happen to know how to mask a region from netcdf file using a shape/netcdf file?

Thanks!
-Kushank

(1-10/10)

Project

General

Profile

CDO