Project

General

Profile

cdo copy/selall changes variable attributes in nc-files

Added by Lina Neunhäuserer about 4 years ago

Hi,

I am trying to zip nc-files with the following command:

cdo -z zip_1 copy file_in.nc file_out.nc

Zipping works fine, but in file_out.nc several variable attributes are missing that are necessary for the further use of file_out.nc.
Using "selall" instead of "copy" leads to the same result. I attach the output of ncdump -c for both file_in.nc and file_out.nc.

Main problem is that
float time_bnds(time, bnds) ;
time_bnds:units = "hours since 2018-01-01 00:00:00" ;
time_bnds:calendar = "proleptic_gregorian" ;
reduces to
double time_bnds(time, nb2)

and
float ASHFLS ;
ASHFLS:units = "W m-2" ;
ASHFLS:long_name = "sensible heat net flux at surface" ;
ASHFLS:positive = "up" ;
reduces to
float ASHFLS ;
ASHFLS:long_name = "sensible heat net flux at surface" ;
ASHFLS:units = "W m-2" ;

in file_out.nc. But it would be nice anyway if a simply "copy" command would just copy...
So is there a way to keep the variable attributes as they are in the original file?

The cdo version I am using here is (cdo -V):
Climate Data Operators version 1.6.4 (http://code.zmaw.de/projects/cdo)
Compiled: by buildd on brahms (x86_64-unknown-linux-gnu) Oct 16 2014 13:09:38
Compiler: gcc -std=gnu99 -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wall -pedantic -fPIC -fopenmp
version: gcc (Debian 4.9.1-17) 4.9.1
Features: PTHREADS OpenMP NC4 OPeNDAP Z JASPER PROJ.4 MAGICS
Libraries: proj/4.8
Filetypes: srv ext ieg grb grb2 nc nc2 nc4 nc4c
CDI library version : 1.6.4 of Oct 16 2014 13:09:17
GRIB_API library version : 1.10.4
netCDF library version : 4.1.3 of Oct 3 2014 13:49:32 $
HDF5 library version : 1.8.13
SERVICE library version : 1.3.2 of Oct 16 2014 13:08:54
EXTRA library version : 1.3.2 of Oct 16 2014 13:08:45
IEG library version : 1.3.2 of Oct 16 2014 13:08:51
FILE library version : 1.8.2 of Oct 16 2014 13:08:45

Best regards, Lina


Replies (11)

RE: cdo copy/selall changes variable attributes in nc-files - Added by Ralf Mueller about 4 years ago

hi Lina!

I think an update to the latest release of CDO (1.9.8) should solve your issue. Is this possible for you?

cheers
ralf

RE: cdo copy/selall changes variable attributes in nc-files - Added by Lina Neunhäuserer about 4 years ago

Hi Ralf,

thanks for your reply. Sorry, it took me some days to install a newer version of cdo.
We now tried CDO 1.9.9rc1. I am sorry to tell that the result is not different from using the old CDO 1.6.4.
The entries missing before are still missing. Any other idea would be appreciated...

Best regards, Lina

RE: cdo copy/selall changes variable attributes in nc-files - Added by Ralf Mueller about 4 years ago

hi!

Ok, do you have a small input file for me? maybe just a single timestep? you could upload it here...

cheers
ralf

RE: cdo copy/selall changes variable attributes in nc-files - Added by Lina Neunhäuserer about 4 years ago

I am sorry but I cannot provide the data to the public. Is there a way to send you the data confidentially?

Best regards, Lina

RE: cdo copy/selall changes variable attributes in nc-files - Added by Ralf Mueller about 4 years ago

I don't need the real values, because the issue is about meta data. In case the meta data can be published, you could set all field values to zero and upload it afterwards.

Could this be an option?

RE: cdo copy/selall changes variable attributes in nc-files - Added by Lina Neunhäuserer about 4 years ago

Yeah, this is an option. Please find the example file attached.

By the way, using
cdo mulc,0 ifile.nc ofile.nc
leads to the missing variable attributes as well.

Best regards, Lina

RE: cdo copy/selall changes variable attributes in nc-files - Added by Ralf Mueller about 4 years ago

hi Lina!

  1. time_bnds inherits the units from time - everything else does not make sense. so it would be redundant to keep these attributes for time_bnds
  2. the variable attribute positive="up" is only allowed to vertical coordinates according to CF-convention. Currently CDO only reads/writes CF-conform attributes, that's why it is lost in any kind of CDO operation. But we plan to review the code in order to leave non-standard attributes for data variables untouched. There is a good chance that this feature will make it into the next CDO release. For now you can re-attach the attribute with
    cdo -setattribute,ALHFLS@positive="up" <input> <output>

Sorry that I don't have a better workaround for you.

cheers
ralf

RE: cdo copy/selall changes variable attributes in nc-files - Added by Fabian Wachsmann about 4 years ago

Dear Lina,

I hope I will be able to test CDOs on conserving meta data with an amount of CMIP6 data, and, eventually, adapt CDO to do so. This includes conserving the 'positive' attribute.

Main problem is that
float time_bnds(time, bnds) ;
time_bnds:units = "hours since 2018-01-01 00:00:00" ;
time_bnds:calendar = "proleptic_gregorian" ;
reduces to
double time_bnds(time, nb2)

This should not be a problem because it is conform to CF. See chapter 7.1:
http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html
"Since a boundary variable is considered to be part of a coordinate variable’s metadata, it is not necessary to provide it with attributes such as long_name and units."

Best regards,
Fabi

RE: cdo copy/selall changes variable attributes in nc-files - Added by Lina Neunhäuserer about 4 years ago

Hi Ralf, hi Fabi,

thanks for your suggestions. Unfortunately, using "cdo -setattribute" produces unzipped output-files. Which means that I have to zip them again which means that the attributes in question are lost again... ...
Besides, using "cdo -setattributes" a second time on a file (to replenish another attribute to another variable) leads to the loss of the previously replenished attribute, not to the replenishment of the second attribute.

Best regards, Lina

RE: cdo copy/selall changes variable attributes in nc-files - Added by Ralf Mueller about 4 years ago

hi Lina!

I don't think so: you can set the attribute and compress the file in one step:

cdo -f nc4c -z zip setattribute,ALHFLS@positive="up" testdata.nc tt.ncz
then the attribute does not get lost.

cheers
rafl

RE: cdo copy/selall changes variable attributes in nc-files - Added by Lina Neunhäuserer about 4 years ago

Hi Ralf,

thank you for this. It works fine as long as you only want to replenish non-standard attributes.
But still we need the time_bnds-attributes. I have been discussing with our project partner but we need that attributes even if they are redundant from your point of view. So now I ended up with the folling two-step approach using cdo and nco (thanks to our project partner):

cdo -z zip_1 copy file_in.nc file_out.nc
ncap2 -Oh -s 'time_bnds@units="hours since 2018-01-01 00:00:00" ; time_bnds@calendar="proleptic_gregorian" ; ALHFLS@positive="up"' file_out.nc file_out_allattr.nc

Best regards and thanks for all, Lina

    (1-11/11)