Project

General

Profile

Concatenate 2 netcdf files with time interpreted by cdo as 0000-00-01

Added by Pawel Pawel about 3 years ago

Hello, I'd like to concatenate 2 netcdf files with timeseries data. I've done some trials and now I think I know how to make it work, but my question is why it wouldn't work with my initial netcdf.

Initial netcdf:
$ ncdump -v TIME infile.nc

dimensions:
    LATITUDE = 100 ;
    LONGITUDE = 80 ;
    TIME = UNLIMITED ; // (12 currently)
variables:
    float LATITUDE(LATITUDE) ;
        LATITUDE:UNITS = "DEGREES_NORTH" ;
    float LONGITUDE(LONGITUDE) ;
        LONGITUDE:UNITS = "DEGREES_EAST" ;
    float TIME(TIME) ;
        TIME:UNITS = "hour since 2011-06-01 00:00:00" ; # in second file: TIME:UNITS = "hour since 2011-06-01 12:00:00" ;
         TIME:time_origin = "2011-06-01 00:00:00" ; # in second file: TIME:time_origin = "2011-06-01 12:00:00" 
...
data:
 TIME = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 ;


$ cdo showdate infile.nc
0000-00-01 0000-00-02 0000-00-03 0000-00-04 0000-00-05 0000-00-06 0000-00-07 0000-00-08 0000-00-09 0000-00-10 0000-00-11 0000-00-12

Now when I concatenate 2 files, one starting at 00 hour and another at 12, it doesn't add another 12 hours, from 13 to 24, but duplicates hours 1 to 12:

Snippets of results for 'ncdump -v TIME outfile' of commands:
$ cdo mergetime infile outfile
TIME = 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 11,
12, 12 ;
$ cdo cat infile outfile
TIME = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11, 12 ;
$ cdo copy infile outfile
TIME = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11, 12 ;

Result of 'cdo showdate' for output files produced by cdo cat and copy:
$ cdo showdate outfile.nc
0000-00-01 0000-00-02 0000-00-03 0000-00-04 0000-00-05 0000-00-06 0000-00-07 0000-00-08 0000-00-09 0000-00-10 0000-00-11 0000-00-12 0000-00-01 0000-00-02 0000-00-03 0000-00-04 0000-00-05 0000-00-06 0000-00-07 0000-00-08 0000-00-09 0000-00-10 0000-00-11 0000-00-12
cdo mergetime:
$ cdo showdate outfile_mergetime.nc
0000-00-01 0000-00-02 0000-00-03 0000-00-04 0000-00-05 0000-00-06 0000-00-07 0000-00-08 0000-00-09 0000-00-10 0000-00-11 0000-00-12

To make it work as I need and expected it initially, I run this command for every file, before using any of the concatenating command:

$ cdo setreftime,2011-06-01,00:00:00,hours -settaxis,2011-06-01,01:00:00,1hour -setcalendar,standard infile.nc outfile.nc

After running this command, mergetime, copy and cat seems to work as expected, they create 24h files with proper dates and times.
However, some properties of netcdf file are changed.

Variables in initial netcdf file:

float LATITUDE(LATITUDE) ;
LATITUDE:UNITS = "DEGREES_NORTH" ;
float LONGITUDE(LONGITUDE) ;
LONGITUDE:UNITS = "DEGREES_EAST" ;

changed to:

float LONGITUDE(LONGITUDE) ;
LONGITUDE:axis = "X" ;
float LATITUDE(LATITUDE) ;
LATITUDE:axis = "Y" ;

and:

    float TIME(TIME) ;
        TIME:UNITS = "hour since 2011-06-01 00:00:00" ;
        TIME:time_origin = "2011-06-01 00:00:00" ;

changed to:

    double time(time) ;
        time:standard_name = "time" ;
        time:units = "hours since 2011-6-1 00:00:00" ;
        time:calendar = "proleptic_gregorian" ;
        time:axis = "T" ;

Was there something incompatible with current cdo or netcdf in my initial netcdf files?
Do I have to run setrefaxis and settaxis to make it work, or is there any simpler way? Maybe just change TIME:UNIT string?
Why it changes my initial variables desctiption, that didn't seem to be touched by those commands? Can I keep them the old way somehow?
Is there something wrong with my initial time units: 'TIME:UNITS = "hour since 2011-06-01 00:00:00" ;'?

Regards,
Pawel


Replies (6)

RE: Concatenate 2 netcdf files with time interpreted by cdo as 0000-00-01 - Added by Brendan DeTracey about 3 years ago

Hard to say without some file samples. Edit: shouldn't it be "hours since"?
Could also try running your files through a CF file checker: https://cfconventions.org/compliance-checker.html

RE: Concatenate 2 netcdf files with time interpreted by cdo as 0000-00-01 - Added by Pawel Pawel about 3 years ago

Thank You for Your response. The above compliance checkers says this regarding TIME variable:

------------------
Checking variable: TIME
------------------
WARN: (3): No standard_name or long_name attribute specified
WARN: (3.1): units attribute should be present

Python netCDF4 library shows that this is netcdf3 file format. Could that be the source of my issue?

I've tried to convert file to netcdf4 format using 'nccopy -k' or using nc3tonc4() from Python netCDF4.utils, but nothing changes regarding TIME variable and date, 'cdo showdate' in new file also starts at 0000-00-01 with 1-day timestep.
I'll try to concatenate it 'by hand' in Python and keep the dates as they are or just use cdo setreftime and settaxis.
Regards, Pawel

RE: Concatenate 2 netcdf files with time interpreted by cdo as 0000-00-01 - Added by Brendan DeTracey about 3 years ago

cdo setattribute,time@long_name=time infile outfile
cdo setattribute,time@standard_name=time infile outfile

RE: Concatenate 2 netcdf files with time interpreted by cdo as 0000-00-01 - Added by Pawel Pawel about 3 years ago

cdo setattribute (Warning): Variable >TIME< not found!
cdo setattribute: Processed 1378080 values from 6 variables over 12 timesteps [0.08s 47MB]

Whether I use 'TIME' or 'time' variable, this doesn't seem to work. It also doesn't work for LONGITUDE or LATITIDE. But works on other variables that store actual data.
However, after any 'cdo setattribute' operation, doesn't matter if variable was found or not, time attributes changes to:
float TIME(TIME) ;
    TIME:standard_name = "time" ;
    TIME:units = "day as %Y%m%d.%f" ;
    TIME:calendar = "proleptic_gregorian" ;
    TIME:axis = "T" ;

RE: Concatenate 2 netcdf files with time interpreted by cdo as 0000-00-01 - Added by Brendan DeTracey about 3 years ago

Got it. You will have to add that attribute using nco operator ncatted i.e. create the attribute/value standard_name=time for variable TIME, i.i.e.e ncatted -a standard_name,TIME,c,c,time or ncatted -a standard_name,TIME,c,string,time

RE: Concatenate 2 netcdf files with time interpreted by cdo as 0000-00-01 - Added by Pawel Pawel about 3 years ago

This works, I've also changed some other TIME variable attributes. It seems that uppercase UNITS is incompatible here.
ncatted -a standard_name,TIME,c,c,time -a units,TIME,c,c,"hours since 2011-06-01 00:00:00" -a calendar,TIME,c,c,standard -a UNITS,TIME,d,c, input.nc output.nc

variables:
    float TIME(TIME) ;
        TIME:time_origin = "2011-06-01 00:00:00" ;
        TIME:standard_name = "time" ;
        TIME:units = "hours since 2011-06-01 00:00:00" ;
        TIME:calendar = "standard" ;

Now date is interpreted properly and I can ncrcat, cdo cat or mergetime properly. cdo changes some attribute names, ncrcat does not.
$ cdo showdate outfile_00-24.nc
  2011-06-01  2011-06-02

$ cdo showtime outfile_00-24.nc
 01:00:00 02:00:00 03:00:00 04:00:00 05:00:00 06:00:00 07:00:00 08:00:00 09:00:00 10:00:00 11:00:00 12:00:00 13:00:00 14:00:00 15:00:00 16:00:00 17:00:00 18:00:00 19:00:00 20:00:00 21:00:00 22:00:00 23:00:00 00:00:00

cf-checker also has no complaints regarding TIME variable.
I think I got what I needed. Thank You for Your help.

    (1-6/6)