BUG - setreftime does not update metadata
Added by Jan Vogelsang about 5 years ago
My netCDF4 files have varying time formats so I use setreftime together with settunits and setcalendar to bring them all to the same format:-setreftime,'1970-01-01','00:00:00' -settunits,hours -setcalendar,standard
This works fine, all times get updated accordingly and now use the hours since 1970-01-01 00:00:00 format. However when looking at the metadata of the files it says
"hours since 2000-1-1 00:00:00" (it has been "days since 2000-1-1 00:00:00" before changing the format).
Is this a bug?
Replies (5)
RE: BUG - setreftime does not update metadata - Added by Karin Meier-Fleischer about 5 years ago
Hi Jan,
This works fine, all times get updated accordingly and now use the hours since 1970-01-01 00:00:00 format. However when looking at the metadata of the files it says "hours since 2000-1-1 00:00:00" (it has been "days since 2000-1-1 00:00:00" before changing the format)
they use the changed time but the reference time is unchanged. How can that be? Are there negative time values? Please, upload some example files and your complete cdo call so that we can reproduce the problem.
-Karin
RE: BUG - setreftime does not update metadata - Added by Jan Vogelsang about 5 years ago
Hi Karin,
I have no idea how that can be, that is why I supposed this is a bug.
The two files I uploaded represent a random dataset before and after reformatting the time axis.
The exact command I used: cdo -f nc4 -z zip -setreftime,'1970-01-01','00:00:00' -settunits,hours -setcalendar,standard testdata_before.nc testdata_after.nc.
When running ncdump -h testdata_before.nc I get:time:units = "days since 2000-1-1 00:00:00" ;
For testdata_after.nc I get:time:units = "hours since 2000-1-1 00:00:00" ;
All other metadata seems to be unchanged (apart from history of course)
ncdump -v,time testdata_before.nc gives me the following timestamps:data:
time = 7152.5, 7153.5, 7154.5, 7155.5, 7156.5, 7157.5, 7158.5, 7159.5,
7160.5, 7161.5, 7162.5, 7163.5, 7164.5, 7165.5, 7166.5, 7167.5, 7168.5,
7169.5, 7170.5, 7171.5, 7172.5, 7173.5, 7174.5, 7175.5, 7176.5, 7177.5,
7178.5, 7179.5, 7180.5, 7181.5, 7182.5, 7183.5 ;
while testdata_after.nc gives me:data:
time = 434628, 434652, 434676, 434700, 434724, 434748, 434772, 434796,
434820, 434844, 434868, 434892, 434916, 434940, 434964, 434988, 435012,
435036, 435060, 435084, 435108, 435132, 435156, 435180, 435204, 435228,
435252, 435276, 435300, 435324, 435348, 435372 ;
Both contain the correct timestamps that reflect the format they should be in. However, for testdata_after the metadata is wrong.
434628 in "hours since 1970-1-1 00:00:00" equals 2019-08-01 12:00:00, the same as 7152.5 does for "days since 2000-1-1 00:00:00".
-Jan
| testdata_after.nc (7.36 MB) testdata_after.nc | |||
| testdata_before.nc (7.05 MB) testdata_before.nc |
RE: BUG - setreftime does not update metadata - Added by Karin Meier-Fleischer about 5 years ago
Using CDO version 1.9.7 everything is correct.
cdo -f nc4 -z zip -setreftime,'1970-01-01','00:00:00' -settunits,hours -setcalendar,standard testdata_before.nc testdata_after2.nc
ncdump -h testdata_after2.nc
netcdf testdata_after2 {
dimensions:
time = UNLIMITED ; // (32 currently)
lon = 3600 ;
lat = 1800 ;
variables:
double time(time) ;
time:standard_name = "time" ;
time:units = "hours since 1970-1-1 00:00:00" ;
time:calendar = "standard" ;
time:axis = "T" ;
double lon(lon) ;
lon:standard_name = "longitude" ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
lon:axis = "X" ;
double lat(lat) ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
lat:axis = "Y" ;
float frpfire(time, lat, lon) ;
frpfire:long_name = "Wildfire radiative power" ;
frpfire:units = "W m-2" ;
frpfire:code = 99 ;
frpfire:table = 210 ;
// global attributes:
:CDI = "Climate Data Interface version 1.9.7.1 (http://mpimet.mpg.de/cdi)" ;
:history = "Fri Nov 22 11:08:17 2019: cdo -f nc4 -z zip -setreftime,1970-01-01,00:00:00 -settunits,hours -setcalendar,standard testdata_before.nc testdata_after2.nc\nWed Oct 2 13:30:01 2019: ncks -d time,61,92 GFAS_frpfire.nc /p/project/cjjsc42/martin/gfas_frpfire_Aug2019.nc\nWildfire emission file retrieved from operational MACC D-FIRE product GFAS at ECMWF (exp id g6ek) and processed by FZJ at " ;
:source = "Data downloaded from ECMWF Mars archive (Experiment g6ek, assimilated daily mean data (timestamp 1209))" ;
:institution = "FZJ (Forschungszentrum Juelich GmbH, Germany); ECMWF (European Centre for Medium Range Weather Forecast, Reading, UK)" ;
:Conventions = "CF-1.5" ;
:NCO = "netCDF Operators version 4.7.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
:experiment_id = "g6ek" ;
:title = "GFASv1.2 daily mean fire emission data for MACC forecasts of atmospheric composition" ;
:contact = "Data: Johannes Kaiser (j.kaiser@ecmwf.int), WCS: Olaf Stein (o.stein@fz-juelich.de), Martin Schultz (m.schultz@fz-juelich.de)" ;
:project_id = "MACC" ;
:CDO = "Climate Data Operators version 1.9.7.1 (http://mpimet.mpg.de/cdo)" ;
}
cdo showtimestamp testdata_after2.nc 2019-08-01T12:00:00 2019-08-02T12:00:00 2019-08-03T12:00:00 2019-08-04T12:00:00 2019-08-05T12:00:00 2019-08-06T12:00:00 2019-08-07T12:00:00 2019-08-08T12:00:00 2019-08-09T12:00:00 2019-08-10T12:00:00 2019-08-11T12:00:00 2019-08-12T12:00:00 2019-08-13T12:00:00 2019-08-14T12:00:00 2019-08-15T12:00:00 2019-08-16T12:00:00 2019-08-17T12:00:00 2019-08-18T12:00:00 2019-08-19T12:00:00 2019-08-20T12:00:00 2019-08-21T12:00:00 2019-08-22T12:00:00 2019-08-23T12:00:00 2019-08-24T12:00:00 2019-08-25T12:00:00 2019-08-26T12:00:00 2019-08-27T12:00:00 2019-08-28T12:00:00 2019-08-29T12:00:00 2019-08-30T12:00:00 2019-08-31T12:00:00 2019-09-01T12:00:00 cdo showtimestamp: Processed 1 variable over 32 timesteps [0.01s 12MB].
You don't need to use setunits because setreftime can do it on the fly appending the units to the setting.
cdo -f nc4 -z zip -setreftime,'1970-01-01','00:00:00',1hour -setcalendar,standard testdata_before.nc testdata_after3.nc
ncdump -h testdata_after3.nc
netcdf testdata_after3 {
dimensions:
time = UNLIMITED ; // (32 currently)
lon = 3600 ;
lat = 1800 ;
variables:
double time(time) ;
time:standard_name = "time" ;
time:units = "hours since 1970-1-1 00:00:00" ;
time:calendar = "standard" ;
time:axis = "T" ;
double lon(lon) ;
lon:standard_name = "longitude" ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
lon:axis = "X" ;
double lat(lat) ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
lat:axis = "Y" ;
float frpfire(time, lat, lon) ;
frpfire:long_name = "Wildfire radiative power" ;
frpfire:units = "W m-2" ;
frpfire:code = 99 ;
frpfire:table = 210 ;
// global attributes:
:CDI = "Climate Data Interface version 1.9.7.1 (http://mpimet.mpg.de/cdi)" ;
:history = "Fri Nov 22 11:10:34 2019: cdo -f nc4 -z zip -setreftime,1970-01-01,00:00:00,1hour -setcalendar,standard testdata_before.nc testdata_after3.nc\nWed Oct 2 13:30:01 2019: ncks -d time,61,92 GFAS_frpfire.nc /p/project/cjjsc42/martin/gfas_frpfire_Aug2019.nc\nWildfire emission file retrieved from operational MACC D-FIRE product GFAS at ECMWF (exp id g6ek) and processed by FZJ at " ;
:source = "Data downloaded from ECMWF Mars archive (Experiment g6ek, assimilated daily mean data (timestamp 1209))" ;
:institution = "FZJ (Forschungszentrum Juelich GmbH, Germany); ECMWF (European Centre for Medium Range Weather Forecast, Reading, UK)" ;
:Conventions = "CF-1.5" ;
:NCO = "netCDF Operators version 4.7.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
:experiment_id = "g6ek" ;
:title = "GFASv1.2 daily mean fire emission data for MACC forecasts of atmospheric composition" ;
:contact = "Data: Johannes Kaiser (j.kaiser@ecmwf.int), WCS: Olaf Stein (o.stein@fz-juelich.de), Martin Schultz (m.schultz@fz-juelich.de)" ;
:project_id = "MACC" ;
:CDO = "Climate Data Operators version 1.9.7.1 (http://mpimet.mpg.de/cdo)" ;
}
cdo showtimestamp testdata_after3.nc 2019-08-01T12:00:00 2019-08-02T12:00:00 2019-08-03T12:00:00 2019-08-04T12:00:00 2019-08-05T12:00:00 2019-08-06T12:00:00 2019-08-07T12:00:00 2019-08-08T12:00:00 2019-08-09T12:00:00 2019-08-10T12:00:00 2019-08-11T12:00:00 2019-08-12T12:00:00 2019-08-13T12:00:00 2019-08-14T12:00:00 2019-08-15T12:00:00 2019-08-16T12:00:00 2019-08-17T12:00:00 2019-08-18T12:00:00 2019-08-19T12:00:00 2019-08-20T12:00:00 2019-08-21T12:00:00 2019-08-22T12:00:00 2019-08-23T12:00:00 2019-08-24T12:00:00 2019-08-25T12:00:00 2019-08-26T12:00:00 2019-08-27T12:00:00 2019-08-28T12:00:00 2019-08-29T12:00:00 2019-08-30T12:00:00 2019-08-31T12:00:00 2019-09-01T12:00:00 cdo showtimestamp: Processed 1 variable over 32 timesteps [0.01s 12MB].
BTW your testdata_after.nc has shifted year 2049 instead of 2019!?
-Karin
| testdata_after2.nc (7.36 MB) testdata_after2.nc | |||
| testdata_after3.nc (7.36 MB) testdata_after3.nc |
RE: BUG - setreftime does not update metadata - Added by Jan Vogelsang about 5 years ago
Ok, awesome. I am using version 1.9.3 as this is the newest version that apt can find on Ubuntu. Seems to be a bug in this version then.
I will definitely try it out with version 1.9.8 later today after installing it from source!
And no the year has not shifted, the metadata is just wrong.
Thanks for the advice about removing settunit and adding the additional parameter to settreftime, will integrate this to my workflow!
-Jan
RE: BUG - setreftime does not update metadata - Added by Ralf Mueller about 5 years ago
Jan Vogelsang wrote:
Ok, awesome. I am using version 1.9.3 as this is the newest version that apt can find on Ubuntu. Seems to be a bug in this version then.
I will definitely try it out with version 1.9.8 later today after installing it from source!
You can also use Anaconda or miniconda. 1.9.8 is available